When I installed and ran Python 3.12 in Ubuntu for the first time, I got a Segmentation fault (Core dumped) error. I downloaded the latest release from the official website. and built Python from the source using the below command:
./configure --enable-optimizations --enable-loadable-sqlite-extensions && make && sudo make altinstall
Debug
To debug this error, I used gdb using the below command:
gdb python3.12
The above command opens the gdb console, then using the run command, I got into the interactive shell of Python 3.12 and checked the reason for the segmentation fault (core dumped) error. This error can occur for multiple reasons:
A third-party extension module written in C has crashed.
Calling external code with built-in ctypes module, that crashes.
Something is wrong with the Python installation.
A bug in Python – should be reported.
In my case, it was Python’s standard library readline that was crashing.
Fix
To fix this error, I simply installed the libreadline-dev from the apt using the below command:
sudo apt install libreadline-dev
And, recompiled the python3.12 from source again:
./configure --enable-optimizations --enable-loadable-sqlite-extensions && make && sudo make altinstall
Often we need to do alternate things in every steps of a django template for loop while doing things in a loop in a django template. For example, let’s say you want to create a widget similar to the following image
widget – to show various job types
In this example, The Job Type widget shows a multi-colored checkbox which shows six colors in total. If more than six items in the list then it will cycle through the colors. Thanks to django’s powerful DTL and jinja2 template engine we can accomplish this in multiple ways. Let’s explore a few possible solutions:
Notice, how we are taking advantage of django’s forloop.counter and the divisibleby tags to make sure in each loop we get an unique and different color cycled in a sequence from red, purple, pink…cyan, red, purple, pink …cyan and so on. There are some problems though. The above code is a nightmare to maintain and has a lot of unnecessary repetition in it. Now take a look at the next example
It sure looks cleaner than the previous one isn’t it? The magic here is the tag cycle. It will cycle through the given list of strings as long as the loop continues. Read more about template tags and filter here
We can use the below command to search git log for commits that touches an specific file
git log --all -- deleted_file.example --oneline
Obviously, instead of deleted_file.example we will have to write the exact name/path of the file that we are looking for. So, if we are searching for main.py it will be git log --all -- main.py
This will list all the commits that touched the file but, there is a better alternative in newer version of gits:
Method 2: using diff-filter
The below command filters commits that deleted file using diff-filter from git log
git log --diff-filter=D -- deleted_file.example
It takes advantage of the flag diff-filter. The cool thing about diff-filter is we can also use it to find the commit that first added the file! example:
In this tutorial, we will be discussing how we can leverage the power of Nginx to create a reverse proxy for our PostgreSQL server. This will allow us to access our PostgreSQL database from remote servers. So for example, let’s say you have an app hosted in a vps and you would like to access the app db from your local computer. Then if you setup the nginx reverse proxy on your vps server, you will be able to connect to postgres db directly from your local computer’s Pgadmin4 app by using your vps ip address and port!
Why Nginx?
Nginx will act as an intermediary service to process the client requests. It is also known for better performance since it uses an event-driven architecture to deal with the client requests asynchronously. Nginx can parse and analyze website URLs fast and serve static content. An Nginx reverse proxy also acts as a line of defense for our PostgreSQL server. Configuring a reverse proxy ensures that the identity of our PostgreSQL server remains unknown, so the PostgreSQL server will be safe from port-scanning and enumeration attacks.
How?
You can follow the step by step guide below on how to install the necessary dependencies and get up and running. We are assuming you already have postgresql installed, if you don’t have it. You can easily install it
Install Nginx
Run the following commands to install nginx in ubuntu:
By default, nginx will create a record in the sites-enabled for a virtual host. We can safely unlink it:
unlink /etc/nginx/sites-enabled/default
Add Nginx Server Configuration
Now, let’s update the nginx configuration. First we will create a new folder webapp in the sites-available folder of nginx. Inside this folder we will keep the server config in a file named: db. Of course, you can name them the way you want. So go ahead open up your text editor e.g. vim, nano, emacs, etc. and create the file. In my case, I’m going to use vim:
vim /etc/nginx/sites-available/webapp/db
in the db file, we will write the following server block:
Notice, we told nginx to listen to port 5431. using the allow we can tell nginx, only the ip address mentioned in the configuration can use the reverse proxy to connect & communicate with our postgresql server. So, if you want to connect from your local computer you can put your own ip address here: allow <your_ip_address>; You can allow multiple ip addresses as well using multiple allow
We also tell nginx to deny all. This makes sure, Nginx will deny all the requests coming from any other ip addresses. Later, we set a timeout of 60s and finally pass/forward the request to postgresql server hosted in the localhost port 5432. Depending on your installation you may have to change these address to match yours. Once, the config file is created we can link it to sites-enabled by using symbolic link:
Now, we will have to edit the nginx.conf file. So, open the nginx.conf file in a text editor:
vim /etc/nginx/nginx.conf
Append the following block of code in the existing nginx.conf
stream {
access_log /var/log/nginx/db.access.log;
error_log /var/log/nginx/db.error.log;
include /etc/nginx/sites-enabled/webapp/db;
}
The above stream block tells nginx to include the server block from the mentioned file: /etc/nginx/sites-enabled/webapp/db TCP is the default protocol for the stream context, so we don’t have to mention it.
Test & reload Nginx
Now, we will test the nginx configuration file using the command:
nginx -t
If we don’t get any error and the configuration files are ok. Then we can reload nginx to start serving:
service nginx reload
That’s it! Now, you should be able to access the postgres database from the ip address mentioned in the nginx configuration. If it doesn’t work, make sure you have the 5431port open (Our nginx server was configured to listen for the tcp network requests in the port 5431)
In Day 1, we learned how to install vim, we also learned the basic navigation shortcuts of vim. Today, we will learn a few more basic vim commands.
More Navigation
some more navigation to ease up scrolling
Page Navigation
^U – Go Up Half a Page (CTRL + U)
^D – Go Down Half a Page (CTRL + D)
Cursor Navigation
zt – Put Cursor to the top
zb – Put Cursor to the bottom
zz – Put Cursor to the middle of the screen
Introducing Help – Your best vim buddy
In Normal mode you can use a command to get help. Just type :h It will open up vim’s help text. You can even search for specific help by specifying what you need, for example:
:h navigation
Vim’s help text is rich & tidy. It will help you to get a good grasp of vim’s philosophy. I highly recommend you to read through it.
Vim Operators
Now, let’s learn a few vim operators. Operators are more like commands that you can apply to objects. Below is the list of most commonly used vim operators:
y – Yank: copy the current object in the cursor position (An object could be word, line, block, etc)
d – delete: deletes the current object in the cursor position
p – Paste: paste the yanked/deleted objects from the buffer
x – removes the objects in the cursor position.
c – Change: deletes the object and puts it into insert mode so that you can type.
We don’t have to memorize anything! Notice how the letter speaks the operation! For example, want to change something? use c. To delete a word we will have to use d, Make sense?
That’s it for today! Let’s practice all these commands and get used to them.
🍪 Vim Cookie #1
Try the following in normal mode! Do you see anything familiar? 😉
Ever wondered how Google knows what you are looking for? Or, had a creepy feeling that Google knows more about you than you know yourself? If so, read on!
Download & Backup all your Google Data
Before, I begin, I think it will be fun to know, what data Google already collected about you. Your daily activities in Google Products can be seen here:
You will be able to collect all the data of every Google Services you ever used! So for example, if you have tons of data in your Gmail, Youtube, Google Drive, etc the amount of data might be overwhelmingly large and take a while to archive! So, be sure to select only the information that you are interested in!
Once, the archiving process is done! Feel free to extract it to your local device and explore various folders. Most of them will be in HTML format or, JSON so it should be easy to understand! For example, take a look at a snippet from my map history
I think the data above is self-explanatory! Just notice, how accurately google detected I was in a passenger vehicle i.e. Uber instead of motorcycling or, walking! The confidence level of Google’s Prediction model is 97.95% for Passenger Vehicle, 1.3% for Walking & 0.4% for Motorcycling!
Fascinating isn’t it?
The above snippet is taken from one of the JSON files in my Location History folder of my Google Data Archive. You can explore yours by extracting it on your local device. If anything is confusing feel free to leave a comment and I will try my best to explain those in a later article (if I get the time)
How exactly Google Collected these Data?
This is a long story. But, in a nutshell, whenever you use a Google product/service you are sharing your personal data with Google. Google utilizes all these data using cutting edge technologies, deep learning, AI, machine learning models to feed over these data to generate valuable information. They usually take consents using a pop up whenever you first-time signup for any of Google Service and ask you for permission by briefly telling you what information they will collect from you! But, hardly anyone read those terms & Agreement and click “I accept” straight away!
Google does provide you some control over the data you share with Google. It’s a lot to cover in a single article so, I’m going to do a quick brief tour of all the data & personalization settings of a Google Account that you can tweak to share as fewer data as possible with Google.
Web & App Activity
As you may have already guessed, this is the control of everything you do inside a Google’s app, for example, Google search, Chrome browsing, Android device activities, etc. You can find these settings in the activity controls page: https://myaccount.google.com/activitycontrols
It is possible to turn it off. You simply have to pause the web & app activity monitoring. To do that, go ahead and click on the Toggle button next to the web and app activity. It will show a pop up where you can safely select pause to turn this google feature off. See below gif for a quick demo:
Toggle web and app activity to Off & click on Pause
Locations & Youtube search history
In the next section, you will find locations history. There you will see a list of all your active devices. You can specifically choose which devices google should keep track of and omit the rest. Or, you can turn the whole thing off by using the toggle button!
Youtube search history setting is also the same as this one, just click on the toggle button to pause it as well.
Ad Personalization Settings
Last but not least, ad personalization settings! Go to the following link:
You can turn it off by clicking on the Toggle like below:
Toggle Ad Personalization to Off & click on “Turn Off”
This page also shows your topic interests, based on which google customizes ads for you. Instead of turning ad personalization off, you can also remove any unnecessary interests from there to refine google’s personalized ads!
But, These are not enough!
The above settings should reduce your Google footprint to a certain degree. But Google might still collect information using its products and services. For example, if you use the Google Chrome Browser or Gmail in your Desktop or android/ios devices, there is a chance that Google will get information. In ios, Google doesn’t have as deep link as android devices still, you can restrict the ‘app permissions’ from the settings to get more control over it.
If you are still concerned about Google, you can even get rid of all Google products and switch to alternatives.
For example, Ditch Google Chrome and use more privacy-oriented browsers such as Tor or Firefox. Or, stop using Gmail & switch to proton mail. You can even go one step further and try Duck Duck go instead of Google search engine. I’m not affiliated and also not advocating any of these products. These are merely just a few examples. If you know some better alternatives feel free to comment and let us know!
In this article, I’m going to introduce you to an interesting python module named pylibdmtx! This library is written based on the libdmtx library. You can install it in Ubuntu by using the following command:
sudo apt install libdmtx0a
Now that you have the library installed let’s install pylibdmtx. I’m assuming you already have python 3 installed. If not go to https://python.org/download and install it right away!
Installation
Once, you have python you can go ahead install the pylibdmtx using pip:
And, that’s it! If the above two commands are executed successfully your system is now fully ready to explore Data Matrix!
Cool! isn’t it?
Data Matrix
Before we begin, exploring data matrix, let’s check out Wikipedia for the definition:
A Data Matrix is a two-dimensional code consisting of black and white “cells” or dots arranged in either a square or rectangular pattern, also known as a matrix. The information to be encoded can be text or numeric data. Usual data size is from a few bytes up to 1556 bytes. The length of the encoded data depends on the number of cells in the matrix. Error correction codes are often used to increase reliability: even if one or more cells are damaged so it is unreadable, the message can still be read. A Data Matrix symbol can store up to 2,335 alphanumeric characters.
Sample Data Matrix collected from search engine
Let’s explore this sample using the pylibdmtx library. We can begin our exploration using python console! We will be using Pillow for reading image.
from pylibdmtx.pylibdmtx import decode
from PIL import Image
image_file = "data_matrix.png"
result = decode(Image.open(image_file))
decoded_data = str(result[0].data)
print('Decoded data:', decoded_data)
If you run the above code using the sample image. You will get an output: 01034531200000111709112510ABCD1234 which is indeed the value of that Data Matrix!
The code above is pretty straight forward, as you can see, first we are loading the image file by opening it using Pillow (PIL). then, we are using the decode method from the pylibdmtx library. The decode method returns a list of results. Since, the above image contains only 1 data matrix, the first item of the list is the Decoded object of our data matrix of that input image. This Decoded object contains the data matrix’s top, left pixel co-ordinates, width, height and data of the data matrix. Using print we simply print the string representation of the decoded data of the Data Matrix.
Note that, you can also get the top left pixel location of the data matrix in the image, and width and height of it using the Rect object of the Decoded Object. Let’s say we store the Decoded object like below:
decoded_object = result[0]
# Print Top Left Pixel Positions
print(decoded_object.rect.top, decoded_object.rect.left)
#Print width & height of the Data Matrix
print(decoded_object.rect.width, decoded_object.rect.height)
Create a Data Matrix from Text
Using pylibdmtx, you can also encode string into data matrix! See the below example:
from pylibdmtx.pylibdmtx import encode
from PIL import Image
encoded_data = encode('wasi0013'.encode('utf-8'))
img = Image.frombytes('RGB', (encoded_data.width, encoded_data.height), encoded_data.pixels)
img.save("wasi0013.png")
Here, we are importing the encode method and then using it to encode the text wasi0013. Then, using the encoded_data object we create an image by using Pillow library. After we save it to a file named wasi0013.png
Output:
data matrix for the text ‘wasi0013’
This is just a quick demonstration of the library. For more information regarding this library check out the official github repository here:
Every year, python enthusiasts from all around the world organize a conference to spread knowledge, learn from each other and make lasting connections. Python Community affectionately calls this national python conference as PyCON. PyCON Dhaka 2016 was probably the 2nd PyCON in Bangladesh. I was lucky to attend this one (missed the first one)
In this article, I’m going to share my first PyCON experience with all of you. Before, I begin I would like to share a little bit about me. I think it will help you to walk through the whole story better.
A bit of Background
Being an introvert, I’ve always stayed away from people. In school, college, or, in varsity, I mingled with only a few people…
For example, In the first few semesters of my university years, even my classmates didn’t know much about me as I merely talked with 1-3 friends! Later, the number of people with whom I used to interact increased quite a bit. Just because, The Class Supervisor choose me as a class representative maybe observing my performance in study, punctuality or, sincerity. However, It was still a strictly closed-loop of friends with whom I used to study, learn and talk.
In a sense, I had built a cocoon that kept me away from people.
This was one of the first steps of getting out of my shell and, my very first interaction with the public as a ‘Software Engineer’ who not just codes but also talks!
The event was organized by a group of Python Enthusiasts of Bangladesh. It is one of the largest python community in Bangladesh. This was my maiden talk in front of the public so I was pretty nervous!
It started with a post in Python Bangladesh of Masnun bhaiya calling for speakers for the PyCON Dhaka 2016. One of my friends showed me that post and asked me to choose one topic and submit there! He encouraged me a lot. We discussed a bit and, later I submitted the topic using the event’s Google Form. I was working on a full-time job as a Team Lead of a small python based R&D team at that time. It was a startup of Chittagong that I joined three months before this event. And, the fast-paced environment and tight working schedule made me forget it all together within a few days! I didn’t really expect to get selected for this topic. I even forget to buy a ticket as a spectator! Later, as the registration for the attendees was closed I was regretting not buying any tickets…
Then suddenly 2 days before the conference, I saw another post which confirmed that I was attending the PyCON as a speaker! My topic got selected.. The first thing I did was a phone call to the managing director and let him know I will need leave. He readily accepted and even encouraged me to attend. I was quite thrilled yet, worried as I lacked preparation. I started to create the slides and started planning my session.
I tried my best to prepare myself for the session in that short amount of time. I watched recent PyCON videos again to brush up my knowledge a bit. Most of the audiences were Experienced Software Engineers and many of them were ‘Geeks with many years of experience’ & I consider them as my mentors. I was very very excited to finally meet those people.
Conference Day
I woke up early in the morning for the Morning Prayer and, didn’t sleep afterward on that day. I expected Dhaka roads to be full of traffic! But, it turned out it was much more than my expectation. The roads were indeed full of traffic, 2x more than my imagination!
I was getting sweaty as I was carrying a backpack with a laptop and other accessories. I walked towards my destination for around 2 hours since I couldn’t find any transports. Finally got into a taxi almost forcefully and reached the venue within a few hours.
Finally Met with PyGeeks & Mentors
On the whole road, I was worried and tensed not only because my timing was all messed up due to the traffic but also for the fact that I thought I’m going to miss my favorite PyGeeks sessions. Thanks to Allah, when I reached there the conference was just getting started. The hall was moderately crowded with many known faces… I was observing the people, the excellent talks, and the environment. The atmosphere was one of a kind!
Especially, the friendliness of the PyGeeks exceeded my expectations. Beginners were participating in spontaneous conversations with them. I’m really proud to be a part of this community.
My Session
I was so absorbed in other sessions, I almost forgot about mine! But, right before my session, Masnun bhaiya made me aware of my reduced session length and asked me to plan my slides accordingly. Hence, I gathered myself back and, was thinking about my slides again.
As expected, I was quite nervous but I tried my best to focus on my topic and explain it to my level best.
Phew! Talking in front of the public is one heck of a task. Those who do it effortlessly, I salute you all!
Even though, I thought I won’t be able to cover the whole slides within the short time frame but, I somehow managed it. Later after the session, I answered a few good questions related to the slide while sipping coffee. Overall, it was an amazing experience!
Conclusion
Thanks for reading this far! You might be wondering why all of a sudden I became so nostalgic. Actually, today is the day ‘October 15th, 2019’. And, Three years ago, on this day, I attended a python conference that had one of the biggest positive impacts on my life.
Things have changed a lot in these three years. But I must acknowledge that attending this conference I learned the importance of knowledge sharing. Inspired by Python Bangladesh Facebook Group, I’m working on building a small Chittagong based Python community named ‘Pyholic‘ for arranging local meetups & workshops. We’ve organized a few meetups and workshops so far and hope to do these regularly.
Three years… I really can’t believe It’s been three years already! And, yet we didn’t have any Python Conference since then…
I know it is tough, but I really hope that we as a community make this type of large scale Conference a regular occurrence in Bangladesh!
In this article, I’m going to demonstrate some code snippets that you can utilize to download files from the Internet using Python. Before we begin, you might be wondering why go through all the hassles of writing scripts to download files when you can simply click and download it by opening it on a browser!
Why?
To ease repetitive task. For example, lets say you are browsing a website with tons of download links and you wan’t to download all these files. Now, to do this manually it will consume a lot of your time. You will get bored or frustrated once you do the same repetitive clicks over and over. This might be a good case for you to automate it using script instead of doing it manually.
Intro
I’m assuming you have a strong basic knowledge of python. I’m going to use some python libraries that are available on the python package index (pip). The codes given in this tutorial is written using Python 3 and tested on a Linux machine. Since It is written in python it should work on other Operating Systems as well.
Download Small/ Large File Using Requests
If you are downloading tiny files you can simply use python’s most popular http module called requests. Install it using pip if not installed already. The code will be similar to this:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
downloads a tiny file using requests module of python 3
In the above script, we are downloading the logo of my website and saving it in a file named logo.png This code should work for tiny files. However, if you want to download some massive sized file that can eat up all your ram! Instead, you can stream the file and consume the content in chunk like below:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sometimes some files might require additional headers/cookies. With requests You can easily set headers & cookies. Easiest way to find which headers & cookies to use is by inspecting the network request using a Web Browser’s Developer tool. For example, in chrome if you press ctrl + shift + i and inspect a page. It will open up a chrome debug tool. In the network tab you can inspect the network requests headers. You can right click on the download request and select “Copy as cURL” to copy the headers as is.
Lets take a look at the way we can set headers and cookies using requests.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
downloading file using requests with custom cookies & headers
Combine wget or, cURL with python to download file
In some cases, downloading Some files might be quite trouble some. For example, if the server is not properly configured or, serves file badly, you may get network error or, errors like ChunkEncodingError or, IncompleteReadError. This can happen for multiple reasons such as content length not defined by the server, or the server closes the connection abruptly, etc. To cope up with this challenge, you can take help from robust command line downloaders such as curl/wget etc. Interestingly, Python has a module called wget, that can utilize the wget available on system. The code is as simple as this:
Of course, you will have to install wget first i.e. pip install wget.
Alternatively, you can use pretty much any other command line downloader that are available in your system such as curl, aria2c, axel etc. For example, see the below snippet:
from subprocess import call
document_url = "https://wasi0013.com/wp-content/uploads/2018/11/my_website_logo_half_circle_green-e1546027650125.png"
filename = "logo.png"
call(["curl", document_url, '-H', 'Connection: keep-alive', '-H', 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36', '--compressed', "--output", filename])
I’m using python’s subprocess module to invoke a terminal command to download the file using an external program called curl. I’ve set two custom headers using the -h argument. Also, output filename/path can be specified using the --output argument.
Download files that requires authentication using Python Selenium & requests
Files that requires authentication & dynamic interaction from the user such as clicking on some button to submit complex forms can be very tricky to download with the above mentioned tools. However, we can easily combine selenium python & requests to achieve it. The trick is to authenticate and do all the interaction stuff using python selenium with a webdriver say chromedriver for example. And, then copy the session, cookies from the driver and set it on a requests session and finally download the file. Sounds complicated? Look at the code below:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
selenium + chromedriver + requests combo for downloading file
If you still don’t understand, just leave a comment I will try to help you understand how it works.
BONUS Trick: Downloading PDF File using javascript, selenium, & python combination
The snippet below is for downloading pdf file using Browser’s Print option. In this script, we are automating a chrome browser with chromedriver that will download a pdf by executing a window.print() Javascript command. You will have to first install selenium & pyvirtualdisplay.
pip install selenium pyvirtualdisplay
I’ve added some comments in the code to clarify the process in detail.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Wasi's Odyssey 
