Acing the Art of Distribution: Mastering Python Wheels and Source Builds#

Source Distributions vs. Wheels#

Python packaging typically revolves around two main distribution formats:

• Source Distribution (sdist): A tarball (or zip file) containing the raw source code and instructions on how to build or install it. Sometimes just called “sdist,” it usually includes your Python files, any necessary data or compiled C source, and a minimal file set describing the build process (such as a setup script).
• Wheel: A pre-built distribution format introduced to overcome the limitations of sdists. Wheels are essentially “built” archives containing your Python files in a directory structure that can be installed without needing to run your code’s build or compile logic. A wheel file typically ends with a .whl extension and supports faster installations.

Below is a quick table summarizing key differences:

The Role of setup.py#

Historically, setup.py has been the entry point for Python packaging. This file uses setuptools (or a similar tool) to define metadata (name, version, author) and the layout of your package (packages, scripts, data files). When you run:

1
python setup.py sdist

it packages your library or application code into a source distribution. For wheels, you typically run:

1
python setup.py bdist_wheel

Modern Packaging with pyproject.toml#

In recent years, Python packaging has adopted the pyproject.toml file as a central place for build configuration, thanks to PEP 517 and PEP 518. It’s part of a move toward more standardized, modern packaging. For many projects, you can add the necessary build system requirements to pyproject.toml, potentially eliminating the need for setup.py. Tools like Poetry and Flit encapsulate this new approach, letting you define dependencies and other metadata in a single place.

1
[build-system]
2
requires = ["setuptools>=42", "wheel"]
3
build-backend = "setuptools.build_meta"
4

5
[project]
6
name = "my-awesome-package"
7
version = "0.1.0"
8
description = "An example modern Python package"
9
authors = [
10
  { name="Your Name", email="you@example.com" }
11
]
12
dependencies = [
13
  "requests>=2.20",
14
  "numpy>=1.18"
15
]

This file can then be used by pip to install your project directly from source:

1
pip install .

Project Structure#

One of the simplest ways to organize your project is by following a structure such as:

1
my_project/
2
  ├── my_package/
3
  │   ├── __init__.py
4
  │   ├── core.py
5
  │   └── helpers.py
6
  ├── tests/
7
  │   └── test_basic.py
8
  ├── setup.py
9
  ├── LICENSE
10
  ├── README.md
11
  └── pyproject.toml

In this scenario, my_package is the actual Python package containing your modules (core.py, helpers.py, etc.). The tests directory holds your test suite, and you have the essential packaging files (setup.py, pyproject.toml) sitting at the top level.

Writing a Minimal setup.py#

Your minimal setup.py might look like this:

1
import setuptools
2

3
with open("README.md", "r", encoding="utf-8") as fh:
4
    long_description = fh.read()
5

6
setuptools.setup(
7
    name="my-package",
8
    version="0.1.0",
9
    author="Your Name",
10
    author_email="you@example.com",
11
    description="A simple example package",
12
    long_description=long_description,
13
    long_description_content_type="text/markdown",
14
    url="https://github.com/you/my-package",
15
    packages=setuptools.find_packages(),
16
    classifiers=[
17
        "Programming Language :: Python :: 3",
18
    ],
19
    python_requires='>=3.6',
20
)

This file leverages setuptools.setup() to define all the usual metadata. Notice it references your README file for a long description—this is often considered best practice, since the description becomes available both in PyPI and on your GitHub or similar repository.

Creating a Source Distribution#

Once you’ve set up your setup.py and optionally a pyproject.toml, you can create a source distribution by running:

1
python setup.py sdist

This command generates a file like my-package-0.1.0.tar.gz within a dist folder. You can upload this archive to PyPI or share it privately, and anyone can then install it:

1
pip install my-package-0.1.0.tar.gz

This is the simplest building block of Python distribution: the source tarball. It’s widely supported, building seamlessly on any platform that has the necessary build tools.

Understanding Wheel Files#

Introduced with PEP 427, wheels are the modern standard for Python distribution. A wheel with a .whl extension is essentially a zip archive with a specific internal layout. It carries compiled code (if necessary) and pre-organized Python modules that can be laid out in the target environment’s site-packages directory without running a build process. By skipping the build step, wheels accelerate installation times—from potentially minutes to just seconds, especially if you have complex C/C++ extensions.

Wheel-Making Tools#

To create wheels, you can rely on setuptools in combination with the wheel library:

1
pip install wheel
2
python setup.py bdist_wheel

Or, if you’re using a modern build configured in your pyproject.toml:

1
pip install build
2
python -m build

This second approach automatically generates a wheel and an sdist inside your dist directory. You’ll typically see something like my_package-0.1.0-py3-none-any.whl appear, named according to Python packaging standards. The segment py3-none-any indicates that this wheel is for Python 3 only, has no ABI (application binary interface) constraints, and is platform-agnostic (any).

Building and Installing Wheels#

A simple step-by-step workflow might be:

1. Create or update your setup.py and/or pyproject.toml.
2. Install build and wheel:

   1
   pip install build wheel
   

3. Build your wheel:

   1
   python -m build
   

4. Inside the new dist directory, you’ll find your .whl file.
5. Install the wheel locally to test it:

   1
   pip install dist/my_package-0.1.0-py3-none-any.whl
   

Once installed, your package is fully ready to be imported as any typical Python library. Because it’s pre-built, there’s no need for compilation at install time, making installations drastically faster.

Advantages of Wheels#

Wheels have become the de facto standard, particularly for projects with compiled extensions. Here’s why:

1. Speed: Eliminating the compile step speeds up installation, especially where compiled code is involved.
2. Reliability: Unified, consistent distribution reduces platform-specific build errors for end-users.
3. Access to Continuous Integration: Many libraries publish wheels for Linux, macOS, and Windows automatically via CI pipelines, ensuring coverage across environments.
4. Future-proofing: The Python ecosystem strongly encourages wheel usage, with most major projects distributing them.

Source Distribution Internals#

A source distribution generally includes:

• All your .py files.
• A setup.py (or pyproject.toml).
• A manifest (often MANIFEST.in) that specifies which files to include or exclude.
• Additional data files if needed (images, configuration templates, locale files).

When a user installs an sdist, pip typically unpacks the archive, checks for dependencies, and runs the build process. This might trigger the execution of your setup.py or the build backend specified in pyproject.toml. If your package has compiled modules, it runs gcc, clang, or other compilers locally. That’s where things can fail if the user lacks the right compiler toolchain or necessary library headers.

Best Practices for Source Distributions#

1. Include a descriptive README.md or README.rst.
2. List all runtime dependencies in setup.py or pyproject.toml.
3. Provide optional development extras for installing testing frameworks and dev tools:

   1
   setuptools.setup(
   2
       extras_require={
   3
           "dev": ["pytest", "mypy"]
   4
       }
   5
   )
   

4. Use a LICENSE file to make your package’s license explicit.
5. Keep the build as platform-neutral as possible if you can. If you need platform-specific steps, document them clearly.

Example: Complex Dependencies#

If your library relies on an external C library, you might need to instruct users to install it before building. For instance, imagine a library that uses the zlib compression library. You might:

• Check for zlib in your setup.py.
• Provide a short tutorial or link to OS-specific installation instructions.
• Offer a fallback or skip the relevant feature if the library is missing.

1
import os
2
import sys
3
import setuptools
4
from setuptools import Extension
5

6
zlib_includes = []
7
if os.name == "posix":
8
    # Example path or check
9
    zlib_includes.append("/usr/include/")
10

11
ext_modules = [
12
    Extension("my_package.zlib_wrapper",
13
              ["my_package/zlib_wrapper.c"],
14
              include_dirs=zlib_includes)
15
]
16

17
setuptools.setup(
18
    name="zlibed_package",
19
    version="0.1.0",
20
    description="Example of a package reliant on zlib",
21
    ext_modules=ext_modules,
22
    # ...
23
)

A user installing from source needs to have the compiler and zlib development headers installed. If they satisfy these requirements, the build proceeds; otherwise it fails, or you handle the error gracefully.

The PyPA Specifications#

The Python Packaging Authority (PyPA) develops and maintains many tools and standards. Key documents include:

• PEP 427: Defines the wheel format.
• PEP 440: Describes versioning conventions for Python projects.
• PEP 517/518: Introduce the pyproject.toml-based build system.

These PEPs unify the ecosystem. For example, any PEP 517-compliant build backend can process pyproject.toml to build your package, enabling ease and interoperability.

pyproject.toml and PEP 517/518#

With the shift to pyproject.toml, you can specify not just your build requirements but also your project’s metadata. For instance:

1
[build-system]
2
requires = ["setuptools>=42", "wheel"]
3
build-backend = "setuptools.build_meta"
4

5
[project]
6
name = "advanced-package"
7
version = "1.2.3"
8
authors = [
9
  { name="Alice", email="alice@example.com" }
10
]
11
description = "An advanced Python package with a modern approach."
12

13
[project.urls]
14
Documentation = "https://mydocs.example.com"
15
Bug Tracker = "https://bugs.example.com"
16

17
[tool.setuptools]
18
# Additional setup if needed

Here, build-system.requires indicates the minimum requirements for building your package. Tools like pip read these fields before installation, ensuring your build environment is consistent.

Poetry, Flit, and Other Packagers#

Though setuptools remains the most common build backend, other tools streamline the packaging process:

• Poetry: Offers a one-stop environment for dependency management, packaging, and publishing.
• Flit: Provides a simpler, more minimal approach to building packages.
• Hatch: Another modern project management tool with built-in support for packaging, testing, and environment creation.

By embracing pyproject.toml, you can switch between these tools more easily. For instance, if you started with Poetry but want to transition to Flit for some reason, you usually won’t need to recreate your entire setup from scratch.

Handling Compiled Extensions#

For libraries that contain C/C++ or other compiled code, wheels are essential to reduce user friction. You might compile modules that integrate with external libraries (e.g., OpenSSL, libpng, or custom HPC libraries). The steps for building such extensions typically go into either:

• A setup.py script that adds Extension objects.
• A specialized build backend configured in pyproject.toml.

Users who install from source must compile on their own system. However, if you provide platform-specific wheels (for Linux, macOS, Windows, etc.), users on those platforms can install them directly without building anything.

Platform Tags and Compatibility#

Wheel filenames contain tags denoting Python compatibility and the platform. For example:

1
my_package-1.0.0-cp39-cp39-win_amd64.whl

This suggests:

• The package version is 1.0.0.
• It’s built for CPython 3.9 (cp39-cp39).
• It targets Windows on an AMD64 architecture (win_amd64).

If your package is purely Python-based (no compiled modules), you might see wheels labeled py3-none-any.whl, meaning Python 3, no ABI constraints, any OS or architecture. This is the simplest scenario.

Multi-Wheel Builds for Multiple Platforms#

Libraries containing native code typically produce separate wheels for each major OS and Python version. To do this systematically:

1. Set up continuous integration (CI) pipelines (e.g., GitHub Actions, Travis CI, Azure Pipelines) that run on multiple operating systems.
2. Configure your CI to build wheels for the relevant Python versions (3.7, 3.8, 3.9, 3.10, etc.).
3. Upload or store those wheels in a place where users can access them (usually PyPI).

After building these wheels, your users can install them via pip without compilation. For example:

1
pip install your_native_package

pip automatically detects the platform and tries to fetch the correct wheel. If no suitable wheel is found, pip falls back to an sdist install.

Testing Wheel Integrity#

It’s a good practice to install the wheel on a clean environment before releasing it. A thorough test might involve:

• Creating a fresh virtual environment:

  1
  python -m venv venv_test
  2
  source venv_test/bin/activate
  

• Installing your wheel and running your package’s test suite:

  1
  pip install path/to/your_package-1.0.0-py3-none-any.whl
  2
  pytest --maxfail=1 --disable-warnings
  

If everything passes, your wheel is likely good to go. This final check can catch weird environment or dependency issues that might be invisible during local development.

Versioning and Release Workflow#

Adopt a clear versioning scheme, ideally PEP 440-compliant semantic versioning. For instance:

• 1.0.0a1 for alpha releases.
• 1.0.0rc1 for release candidates.
• 1.0.0 for stable releases.

Include a CHANGELOG.md (or other form of release notes) so users know what changed. Tag your releases in your version control system—this helps track down which commit produced a given version.

Continuous Integration (CI) for Packaging#

It’s common to automate building and uploading wheels using CI. For instance, with GitHub Actions, you can define workflows triggered on new tags to:

1. Install build dependencies (Python, pip, setuptools, wheel).
2. Build wheels on multiple platforms (Windows, Ubuntu, macOS).
3. Run tests.
4. If successful, upload wheels to PyPI using a secure token.

Below is a simplified example of a GitHub Actions workflow snippet in .github/workflows/publish.yml:

1
name: Publish to PyPI
2
on:
3
  push:
4
    tags:
5
      - 'v*.*.*'
6

7
jobs:
8
  build-and-publish:
9
    runs-on: ubuntu-latest
10
    steps:
11
      - uses: actions/checkout@v2
12
      - name: Set up Python 3.x
13
        uses: actions/setup-python@v2
14
        with:
15
          python-version: '3.x'
16
      - name: Install build dependencies
17
        run: |
18
          pip install build twine
19
      - name: Build
20
        run: |
21
          python -m build
22
      - name: Publish
23
        run: |
24
          python -m twine upload dist/*
25
        env:
26
          TWINE_USERNAME: __token__
27
          TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}

Here, we only show a single environment. In practice, you could run a matrix build that covers multiple Python versions and operating systems.

Releasing on PyPI and Private Repositories#

Public libraries generally go to the Python Package Index (PyPI). However, if your code is proprietary or you have internal packages, you might push them to a private repository, for instance using a self-hosted package index or tools like Artifactory or Nexus that support PyPI proxying.

Publishing to PyPI typically follows a sequence like:

1
python -m build
2
python -m twine upload dist/*

where twine is a secure upload utility. You must have a PyPI account and generate an API token.

Handling Large Data Files and Extras#

Sometimes your package includes large data files or optional dependencies. If your data is massive, consider:

• Hosting it externally and letting your package download it on first run.
• Providing an extra “data” or “heavy” installation variant with an additional dependency or data file distribution.

For optional dependencies, you can define them under extras in setup.py or pyproject.toml. For instance:

1
setuptools.setup(
2
    name="my_package",
3
    version="0.2.0",
4
    install_requires=["requests"],
5
    extras_require={
6
        "images": ["Pillow"],
7
        "dev": ["pytest", "flake8"]
8
    }
9
)

Then, a user can install with extras:

1
pip install my_package[images]

Professional-grade packages often have multiple extras for sub-features, integrating well with specialized subsets of users.