We’ve all been there. Our code is running a job, and it’s going to take a while. Scratching our head, we pray that our code isn’t stuck in some infinite loop. When dealing with large data sets, however, even the simplest operations can take hours to complete. After all, a watched pot never boils, and a watched loop never breaks
Why Use a Python Progress Bar?
If you didn’t guess already, I am an impatient guy. Using an ETA or progress bar while executing code works wonders for my anxiety.
Fortunately, there are libraries out there that provide this functionality using only a couple of lines. In this tutorial, I will introduce four libraries and compare them based on a variety of factors.
How to Set Up a Progress Bar in Python
The code used in this tutorial was tested on an Ubuntu system with Python 3.6.9 installed.
I would highly recommend setting up a virtual environment with all the required libraries for testing. Here is how you can do it.
$ virtualenv progress_env $ source ./progress_env/bin/activate $ pip3 install progress progressbar2 alive-progress tqdm
The four libraries that we are going to work with include:
Progress Bar Using Progress
Progress is a fairly simple and intuitive library. It provides a wide variety of styles to pick from. All progress bars can be integrated into our loops using either a context manager or by wrapping up an iterable object into a method. Let’s take a look at some of them.
A simple progress bar that is filled with hash.
from time import sleep from progress.bar import Bar with Bar('Processing...') as bar: for i in range(100): sleep(0.02) bar.next()
We first import the bar class from the progress.bar module and create its object. We supply the prefix
'Processing...' that will be added to the front of our progress bar. To update the progress bar, we use the
next() method at the end of each iteration.
There are multiple arguments available to us like
suffix. Let’s try modifying the fill from
@ and also look at the ETA.
from time import sleep from progress.bar import Bar with Bar('Loading', fill='@', suffix='%(percent).1f%% - %(eta)ds') as bar: for i in range(100): sleep(0.02) bar.next()
There are times when we don’t know how long an operation is going to take. In such cases, we can use spinners to visualize the process rather than progress bars.
from time import sleep from progress.spinner import MoonSpinner with MoonSpinner('Processing…') as bar: for i in range(100): sleep(0.02) bar.next()
Progress Bars Using Progressbar2
Python-Progressbar (aka Progressbar2) is an extremely popular and easy to use module. It provides some very powerful features like auto-resizing. You can control the format of the progress-bar in the form of widgets like “AbsoluteETA” and “AdaptiveETA.”
Let’s take a look at a basic use-case:
from time import sleep from progressbar import progressbar for i in progressbar(range(100)): sleep(0.02)
Although in the beginning we installed the library as progressbar2, it will be imported as progressbar.
Combining Progress-Bars With Print Output
One thing that we didn’t discuss earlier with the progress library: What happens when the for loop contains a print function?
from time import sleep from progress.bar import Bar with Bar('Processing',max = 5) as bar: for i in range(5): sleep(0.1) print('\n',i) bar.next()
As you can see, this will make the progress bar very messy, and sadly, there is no workaround in the progress library for this. However, Progressbar2 allows us to redirect standard output.
from time import sleep from progressbar import progressbar for i in progressbar(range(100), redirect_stdout=True): print('Some text', i) sleep(0.1)
Building Animated Progress Bars With Alive-Progress
If you love animations and you want to make your progress bars look interactive, then look further. This library contains some of the coolest looking progress-bars available. Take a look:
(Left: Alive-Progress Available Progress-Bars, Right: Alive-Progress Available Spinners (Source: Documentation))
In terms of code, it’s pretty similar. We will present a default progress-bar, and then one with a couple modifications:
from alive_progress import alive_bar from time import sleep with alive_bar(100) as bar: # default setting for i in range(100): sleep(0.03) bar() # call after consuming one item # using bubble bar and notes spinner with alive_bar(200, bar = 'bubbles', spinner = 'notes2') as bar: for i in range(200): sleep(0.03) bar() # call after consuming one item
As you can see above, we get information regarding ETA and iterations per second along with the progress bar. The Github documentation includes details regarding some really powerful features that this library has.
Progress Bars inTQDM
If you need a no-bullshit, speedy yet powerful progress-bar, look no further than TQDM.
TQDM provides a fast framework with a lot of customization and information. Even with all the features, it’s still extremely simple to set up. Take a look:
from tqdm import tqdm from time import sleep for i in tqdm(range(100)): sleep(0.02)
By default, you get info such as percentage, ETA and even iterables per second. TQDM works on all major platforms like Linux, Windows and Mac. It integrates seamlessly in a console, a graphical user interface (GUI) and even in IPython/Jupyter notebooks.
IPython/Jupyter is supported via the
tqdm.notebook sub-module. Let’s take an example with nested progress bars:
This feature with multiple progress-bars is unique to TQDM.
There’s also a pandas integration:
import pandas as pd import numpy as np from tqdm import tqdm df = pd.DataFrame(np.random.randint(0, 1000, (100000, 600))) tqdm.pandas(desc="my bar!") df.progress_apply(lambda x: x**2)
TQDM also has an experimental GUI-Version called tqdm-gui.
TQDM is an extremely important project for individuals working in data science.
Advantages of Progress Bars
Progress bars provide us with:
- A visual cue that processing is underway.
- A reliable estimate of the code execution time. This is extremely important when training on large data sets.
- Indication if the program is stuck in an error and must be stopped.
TQDM has the lowest overhead of all the libraries discussed above at about 60 ns per iteration. By comparison, the well-established ProgressBar2 has an 800 ns/iter overhead.
Most of these libraries don’t have any dependencies.
Progressbar2 and alive-progress officially support print statements along with the progress bar.
In this tutorial, we saw how with only a few lines of code, you can implement progress bars in your python scripts.