ProbableOdyssey

Old Hat, New Tricks: Makefiles for Python Projects

Makefiles have been a staple of the C/C++ toolchain for decades. In a nutshell: GNU Make executes discrete steps that are defined in a Makefile. The steps defined in a Makefile can have dependencies, so that when a step is executed, all the dependent steps are executed first. This enables effective version control and cleaner documentation for the manual steps that are needed to build and run a project.

Like many GNU utilities, it’s been around for a while and is easily overlooked by developers until they encounter a project that uses them effectively. I recently had that experience in some recent Python repositories I’ve been working in. At first I was skeptical, but there are many benefits to using them in modern projects that I didn’t previously consider:

Previously I used a stack of bash scripts to achieve the same goal. But I found that there’s a lot of careful organisation needed to avoid user error caused by executing steps in an incorrect order. Now I find that with using the Makefile, much shorter documentation is needed – as all the details and the dependencies between steps are handled in the construction of the Makefile. The steps in make can also execute any shell script, so steps can still be written in scripts and orchestrated with make.

One sneaky “gotcha” with writing a Makefile is that the steps require indentation with one tab character, not spaces (I found this blog post, which contains an email from the author of make who regrets this decision with good humor!). I use this syntax file to handle this automatically when writing a Makefile in vim:

1" ~/.vim/ftplugin/make.vim
2set noexpandtab shiftwidth=8 softtabstop=0

So why is it useful in Python? There are so many ways to manage dependencies in Python (and of course, none of them quite meet the 100% mark). Now instead of meticulously documenting how to use the tool of choice for a project, I simply document which tool is needed and let the Makefile contain all the details.

For example, here’s a Makefile I would write for a Python project that uses poetry for dependency management, pytest for testing, ruff for linting and formatting, and docker for building images:

 1SHELL := /usr/bin/env bash
 2.SHELLFLAGS := -eu -o pipefail -c
 3
 4
 5.PHONY: _check-python-version, install-uv, install-py-deps, install-py-deps-dev, check-tests, check-lint, check-format, check-all, build-image, build-image-dev, load-dev-shell
 6
 7
 8_check-python-version:
 9    @version=$$(python --version 2>&1); \
10        if [ "$$version" != "Python 3.13.7" ]; then \
11        echo "Error: Python version is not 3.12.7. Detected version: $$version"; \
12        exit 1; \
13    fi
14
15install-poetry: _check-python-version
16    @brew install pipx
17    @pipx install poetry==1.8.4 --python=$(which python)
18
19install-py-deps:
20    @poetry install --sync --only=main
21
22install-py-deps-dev:
23    @poetry install --sync --only=main,dev
24
25check-lint:
26    @poetry run pytest
27
28check-lint:
29    @poetry run ruff check
30
31check-format:
32    @poetry run ruff format --check .
33
34check-all: check-tests check-lint check-format
35
36build-image-base:
37    @docker build -t "my-container-base" -f Dockerfile.base .
38
39build-image-dev: build-image
40    @docker build -t "my-container-dev" -f Dockerfile.dev .
41
42load-dev-shell: build-dev-image
43    @docker exec -it "my-container-dev" bash

The targets in .PHONY ensure that the steps execute every time they’re called, and @ suppresses the command output. Using this file, all my steps are version controlled, unwieldy flags and options for commands don’t need to be copy-pasted, and users with brew can get up and running with a few simple commands:

1$ make install-poetry
2$ make install-py-deps-dev
3$ make check-all
4$ make load-dev-shell

The Makefile is here to stay. It’s simplified my development process, and it’s now one of the first things I’ll write for a new project

References:

Reply to this post by email ↪