This is kind of obvious, but I think it is often skipped over for more complicated solutions: you can pass data from module-to-module using imports. This approach to modular design via modular programming allows us a lightweight way to selectively execute code modelled as a directed acyclic graph (DAG).
Toy Example
Let’s look at a small example involving two modules foo and bar.
./posts/0001-python-modules-can-pass-data-too/simple_example/
├── bar.py
└── foo.py
1 directory, 2 files
In this example we load data into foo.
"""Load the foo data."""
# INFO: Load dependencies
import json
import pathlib
# INFO: Configuration file IO
FILE_PATH = pathlib.Path("data.json")
MODE = "r"
ENCODING = "utf-8"
# INFO: Configuration data validation.
EXPECTED_FIELD = "datasetName"
# INFO: Open the file safely using a context manager
with FILE_PATH.open(MODE, encoding=ENCODING) as file:
data = json.load(file)
# NOTE: We can put data validation right after loading for fail-fast data validation.
if not data.get(EXPECTED_FIELD):
msg = f"dataset must have {EXPECTED_FIELD}?"
raise KeyError(msg)And let’s define bar as a module which does some data processing.
"""Calculate statistics from foo data."""
import foo
ages = [_.get("age") for _ in foo.data.get("users")]
mean_ages = sum(ages) / len(ages)Although it is a toy example, this illustrates a way of building out a Python data science project using module imports. We could build out various modules to be responsible for different things, and pass data through imports accordingly.
Language Server Protocol (LSP) Support
One of the extrinsic aspects of this approach that I enjoy is the opportunity to lean heavily into language server protocol (LSP) support in editors. Want to find where a Python variable came from in your project? Use LSP go to definition. Want to find, or look through, all the places where a Python variable is used? Use LSP go to reference. Want to rename a variable consistently throughout the code? Use LSP rename.
Module Dependencies
LSPs will allow you to efficiently navigate with respect to a given variable under your cursor.
Imports form a dependency graph in the form of a directed acyclic graph. There are tools for extracting this structure. A project still being maintained for over a decade is the pydeps Python package.
It can provide a bird’s-eye view of how modules are related to each other via imports. There is also the ruff analyze graph which is currently experimental but released for use. If you’re using ruff anyway, you can also use the include-dependencies to explicitly map out dependencies that are not visible via imports.
Full(er) Example
Let’s look at a more involved example now using a design which will accommodate the project becoming large.
./posts/0001-python-modules-can-pass-data-too/spaceflights_pandas/
├── pyproject.toml
├── README.md
├── src
│ ├── spaceflights_pandas
│ │ ├── config
│ │ ├── datasets
│ │ │ ├── companies.py
│ │ │ ├── __pycache__
│ │ │ │ ├── companies.cpython-313.pyc
│ │ │ │ ├── reviews.cpython-313.pyc
│ │ │ │ └── shuttles.cpython-313.pyc
│ │ │ ├── README.md
│ │ │ ├── reviews.py
│ │ │ └── shuttles.py
│ │ └── pipelines
│ │ └── data_processing.py
│ └── spaceflights_pandas.egg-info
│ ├── dependency_links.txt
│ ├── PKG-INFO
│ ├── requires.txt
│ ├── SOURCES.txt
│ └── top_level.txt
├── tests
│ └── datasets
└── uv.lock
10 directories, 16 files
