Dynammically importing modules in Python

Use case #

Before we delve into the code, let’s discuss a use case for dynamically importing modules in Python.

Consider a program that will generate maths questions for students to solve. There will be a group of questions for a a particular type. e.g. dice rolling probability, triangle area, etc. A module for each type of question will be created.

The program will be command line driven. The user will specify the type of question they want to generate specified by the first argument.

python3 -m testpapers dice

Code structure #

The main project folder is called testpapers containing the entry point __main__.py script and a subdirectory called test_builders containing the modules that will generate the question papers.

testpapers/
├── __main__.py (the entry point)
└── test_builders/
    ├── __init__.py
    ├── dice.py (module to generate dice questions)
    └── triangle.py (module to generate triangle questions)

The program takes the type of question as an argument to create a list of questions for that type. The __init__.py file is required to make the test_builders directory a package. The dice.py and triangle.py script files will contain the code to generate the questions for each type. These each have a function generate_questions() that will be called from the __main__.py script.

As more types of questions are added, new modules will be created in the test_builders directory.

Problems with the standard import approach #

Typically you would import all the modules at the start of your program, and depending on the question type, call the appropriate function to generate the questions.

import dice
import triangle
# and more imports...

def generate_test_paper(question_type):
    if question_type == 'dice':
        dice.generate_questions()
    elif question_type == 'triangle':
        triangle.generate_questions()
    # and more question types...            
    else:
        print(f"Unknown question type: {question_type}")

Some problems with this approach are:

• All the modules are imported at the start of the program, which can be cumbersome to manage.
• As each module is imported, the code in the module is executed which can be expensive if the modules are large or complex.
• If the modules log progress and diagnostics, a lot of unnecessary output is produced to wade thorugh.
• If the modules have side effects (e.g., modifying global variables, opening files, etc.), these side effects will occur even if the module is not used.

The importlib approach #

The importlib is a built-in Python module that provides a way to import modules programmatically. This can be useful for loading modules at runtime based on certain conditions or configurations.

Let Python find the modules #

However, the importlib module does not provide a way to import modules from a subdirectory.

The sys.path list is used by Python to search for modules when you use the import statement.

To do this, we need to add the subdirectory test_builders to the sys.path list, which is a list of directories that Python searches for modules. By default, it includes the current directory and the standard library directories.

We can add our subdirectory to the sys.path list using the sys.path.append() method.

First we should use the import test_builders as builders, so that it is easier to refer to the directory, and then we can add the test_builders directory to the sys.path list.

import sys
import test_builders as builders

def main():
    sys.path.append(builders.__path__[0])

The __path__ attribute of a module is a list of directories that the package can be found in. It is a list as the package can be spread in multiple directories, but example sub-packages. In this case, we are only interested in the first element of the list, which is the directory where the test_builders package is located.

An example of the builders.__path__ list is:

['/home/user/projects/testpapers/test_builder']

Dynamically import the module #

Next, we can use the importlib module to import the module. For this, we can use the importlib.import_module() function. This function takes the name of the module as a string and returns the module object.

import importlib
import logging 


def generate_test_paper(module_name):
    try:
        module = importlib.import_module(module_name)
    except ImportError as err:
        # log the error 
        print(f"Error importing module '{module_name}': {err}")
        return

If the module is found it be available in the module variable, if not a ImportError will be raised, and we print an appropriate error message.

Note that the ImportError can be raised for a number of reasons, including:

• The module does not exist
• The module is not in the sys.path list (if you have not added the subdirectory to the sys.path list)
• The module has a syntax error or other error that prevents it from being imported

Now that we have the module object, we can call the generate_questions() function from the module.

You can also check that the module has the function before calling it by using the hasattr() function on the module object

    # check if the module has the generate_questions() function before calling it
    if hasattr(module, 'generate_questions'):
        module.generate_questions()
    else:
        print(f"Module {module_name} does not have a generate_questions() function")