Guide to Python Logging

Imagine you are working on software consisting of lines and lines of code with a constrained time for submission and all of the sudden, your program crashes. The probability of detecting the reason for the crash is very slim and doing so is time-consuming, too. 

The ideal way to avoid such situations is to track events when the software runs. This is what logging does.

Logging is the process of capturing and storing data about events and activities that occur within a software application or system. Python has a built-in logging module that allows a flexible framework to create log messages in programs. It allows you to record information about events that occur during program execution, such as errors, warnings and debugging messages. 

This article will detail the benefits of logging and how to get started with logging in Python, as well as a brief introduction to some of the advanced features available in the logging module.

More from Sean BenhurGuide to Meta Learning

Benefits of Logging

In adding to saving you time and trouble, logging has many benefits. 

• Logging useful data at the right places helps debug errors easily and, in the case of complex programs, is more efficient than printing.
• It helps a developer better understand the flow of a program and keep an eye on the application.
• It gives more perception in case an error occurs than traceback methods by providing the details of the state of the program before arriving at the line of error. 
• It can also be used to save information on warnings, errors, and other events while executing a program.
• Logs can help analyze the history of the program, providing insights, patterns and trends.

Standard Library Logging Module

Python’s built-in logging library can be used for logging. It is a flexible and powerful framework for logging messages from the Python application. It can be used to log messages on different destinations such as consoles, files or network sockets. It has comprehensive and flexible features that developers can log messages in any of their Python applications.

Some of the features of the Python logging module are:

• Loggers, handlers and formatters. These functions in the module help to log the messages in a proper way such as how the log messages are displayed, where it is saved. 
• Logging levels. The logging module provides different levels which can be used to understand the severity of logging.
• Contextual logging. The logging module supports adding contextual information to logging messages such as user-id, and session-id. So when we read the log messages we can understand them easily.
• Hierarchical loggers. The logging module supports a hierarchical logger namespace, where loggers can inherit settings and handlers from their parent loggers. This feature helps to use the logger when our application consists of many different modules.

Let’s look at a very basic example of logging in Python.

import logging
log = logging.getLogger("user")
log.error("It's an error")

In the above example, we register the “user” with the logger and log the error message on the severity level of “error”.

Levels of Log Messages:

The logging module offers five levels that specify the severity of events. Each level contains parallel methods which can be used to log events based on their severity. The five levels along with the integer value represent the severity of the log.

The example for setting the logging level is:

import logging

# Create a logger
logger = logging.getLogger('my_logger')
logger.setLevel(logging.DEBUG)

In the above code, we set the logging level to DEBUG, which is able to log the messages on the program from and above the DEBUG level.

Basic Logging Configurations

The logging module in Python provides a flexible way to manage and output log messages from the Python code. The basic configuration of logging includes three main components: loggers, handlers and formatters.

1. Loggers. A logger is responsible for emitting log messages from the code. You can create multiple loggers to separate logs for different parts of your application. Loggers are organized hierarchically and identified by name, a string identified by the dots. When a logger is created, it is given a name that identifies its location in the logger hierarchy.
2. Handlers. A handler defines where the log messages will be saved. A logger can have multiple handlers, which allows you to send log messages to different locations such as the console, a file, or a network socket.
3. Formatters. A formatter defines the layout of the log messages differently, depending on where they are being outputted.

Here’s an example of the basic configurations of logging. 

import logging

# Create a logger
logger = logging.getLogger('my_logger')

# Create a handler and set its level
handler = logging.StreamHandler()
handler.setLevel(logging.DEBUG)

# Create a formatter
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

# Add the formatter to the handler
handler.setFormatter(formatter)

# Add the handler to the logger
logger.addHandler(handler)

# Set the logger level
logger.setLevel(logging.DEBUG)

Once the above code is used on any application, only the levels after the DEBUG will be displayed on the console.

More Reading About PythonHow to Make a Python Calculator

Advanced Python Logging Module

Python’s logging module has multiple advanced features which can be used for different use cases. Among the advanced features: 

Loggers With Different Configurations

You can create multiple loggers with different configurations for different parts of the application. Here’s an example of two loggers with different configurations.

import logging

# Create a logger for the application
app_logger = logging.getLogger('myapp')
app_logger.setLevel(logging.INFO)

# Create a logger for a specific module
module_logger = logging.getLogger('myapp.module')
module_logger.setLevel(logging.DEBUG)

Filters

Filters can be used to selectively exclude or include certain log messages based on certain criteria. If we only want to keep a record of messages that contain a particular string, or if we want to exclude messages that fall below a specific logging level, we can use filters.

Custom Handlers

The logging module provides several built-in handlers such as StreamHandler and FileHandler. Based on the needs, we can write a custom handler as well. Here’s an example of the custom handler that outputs the log messages to the MongoDB database.

import logging
from pymongo import MongoClient

class MongoDBHandler(logging.Handler):
    def __init__(self, host, port, db, collection):
        super().__init__()
        self.client = MongoClient(host, port)
        self.collection = self.client[db][collection]

    def emit(self, record):
        message = self.format(record)
        self.collection.insert_one({'message': message})

In the above code, we create a custom handler MongoDBHandler that inherits from the logging handler. The __init__ method takes the host, port, database and collection as arguments and sets up a connection to the MongoDB database. The emit method takes a logging record and inserts the formatted messages into the specific database.

RotatingFileHandler

You can use a RotatingFileHandler to rotate log files based on size or time. Here, rotating refers to the process of creating a new log file and moving the old file to a new backup location when the file reaches a specified criteria either in size or time. An example: 

# Create a rotating file handler that rotates log files after they reach 1 MB
rotating_handler = logging.handlers.RotatingFileHandler('myapp.log', maxBytes=1024*1024, backupCount=5)

# Add the handler to the logger
logger.addHandler(rotating_handler)

In the above code, we create a RotatingFileHandler that rotates log files after they reach 1 MB in size, with up to five backup files. The maxBytes argument specifies the maximum size of each log file, and the backupCount argument specifies the number of backup files to keep.

Logging to Multiple Destinations

You can configure a logger to output log messages to multiple destinations such as a file or a network socket. This can be useful if you want to log files locally while also sending real-time log messages to the remote server.

Logging is a crucial aspect of any software development process. Python’s logging module provides a different set of features that can be customized according to the user’s needs.