Crack the Packaging Puzzle: Build, Distribute, and Deploy Python Apps#

Modules#

A Python “module” is a single file of Python code, typically ending with .py. When you import something, Python searches for a file named something.py or a folder named something containing __init__.py.

Example contents of a simple module calculator.py:

1
def add(a, b):
2
    return a + b
3

4
def subtract(a, b):
5
    return a - b

You can use it in another file by writing:

1
import calculator
2

3
print(calculator.add(5, 7))       # Output: 12
4
print(calculator.subtract(10, 3)) # Output: 7

Packages#

A Python “package” is a directory containing a special file called __init__.py. This file can be empty, or it can define the package’s attributes. Inside that directory, you might have multiple modules (i.e., multiple .py files), subdirectories, and subpackages.

Example directory structure:

1
my_app/
2
    __init__.py
3
    utils.py
4
    main.py
5
    submodule/
6
        __init__.py
7
        data_processing.py

To import something from data_processing.py, you can do:

1
from my_app.submodule.data_processing import transform_data

Or if my_app/submodule/__init__.py imports it:

1
from my_app.submodule import transform_data

In essence, modules and packages help organize your code logically. Packaging is about ironing out how to distribute that organized code as a cohesive unit.

Installing Virtual Environment Tools#

If you use Python 3.3 or later, you already have the built-in venv module. Create a new virtual environment with:

1
python -m venv venv

Then activate it:

• On macOS/Linux:

  1
  source venv/bin/activate
  

• On Windows:

  1
  venv\Scripts\activate
  

You’ll then see (venv) prepended to your shell prompt, indicating you’re inside the virtual environment.

Installing Dependencies#

Once the environment is active, any libraries you install with pip or any other tool will go into that virtual environment.

1
pip install requests

This also means you’ll have a fresh start for each project, preventing version conflicts between unrelated projects.

The setup.py File#

Your setup.py file typically uses setuptools to define package metadata (name, version, author, etc.). Here’s a minimal example:

1
from setuptools import setup, find_packages
2

3
setup(
4
    name="mathlib",
5
    version="0.1.0",
6
    description="A simple math library",
7
    author="Your Name",
8
    author_email="you@example.com",
9
    packages=find_packages(),
10
    install_requires=[],
11
)

• name: This is how your package will be listed on PyPI (if you distribute it there).
• version: Follows semantic versioning or another scheme that you prefer.
• packages: find_packages() automatically detects Python packages.
• install_requires: The list of dependencies your project needs.

Structuring Your Package#

In the mathlib/calculations.py module, you might have:

1
def multiply(a, b):
2
    return a * b
3

4
def divide(a, b):
5
    if b == 0:
6
        raise ValueError("Cannot divide by zero.")
7
    return a / b

The __init__.py could import these so they’re accessible at the package level:

1
from .calculations import multiply, divide

With this structure, anyone can import your package after installation with:

1
import mathlib
2

3
result = mathlib.multiply(3, 4)  # 12

README and LICENSE#

While not strictly required for building a package, having a README.md is best practice, especially if you plan to distribute your package publicly. A proper license file is also recommended.

Packaging Tools Overview#

Specifying Requirements#

Dependencies are typically listed in install_requires within setup.py or another configuration file. For example:

1
setup(
2
    ...
3
    install_requires=[
4
        "requests>=2.20.0",
5
        "numpy==1.21.0",
6
    ],
7
    ...
8
)

When someone installs your package, pip will ensure those dependencies are also installed.

requirements.txt vs install_requires#

• requirements.txt: A common approach for pinning dependencies in an application, used by pip install -r requirements.txt.
• install_requires: For libraries intended for distribution, it’s safer to specify broader version ranges so as not to cause conflicts.

Extras#

If your package has optional features (e.g., a “dev” set of tools), you can use “extras” in setup.py:

1
setup(
2
    ...
3
    extras_require={
4
        "dev": ["pytest", "flake8"],
5
        "docs": ["sphinx"],
6
    },
7
)

Then users can install them with:

1
pip install mathlib[dev]

Test PyPI vs Production PyPI#

It’s advisable to test your upload process on Test PyPI first. Test PyPI is a separate instance of PyPI that lets you experiment with package uploads without polluting the main index.

Steps to Upload#

1. Register an account on PyPI and Test PyPI: Use the same username on both if possible.
2. Edit your ~/.pypirc (optional but helpful for storing credentials).
3. Build your distributions: python setup.py sdist bdist_wheel.
4. Upload with twine:

   1
   twine upload --repository-url https://test.pypi.org/legacy/ dist/*
   

   Once you’re confident, upload to the real PyPI:

   1
   twine upload dist/*
   

Installing from PyPI#

If you publish mathlib to PyPI, then any user can install it by:

1
pip install mathlib

Installing Poetry#

Install Poetry via their recommended script:

1
curl -sSL https://install.python-poetry.org | python3 -

Creating a New Project#

1
poetry new mathlib

This generates a structure like:

1
mathlib/
2
  pyproject.toml
3
  README.md
4
  mathlib/
5
    __init__.py
6
  tests/
7
    __init__.py
8
    test_mathlib.py

Managing Dependencies with Poetry#

From within the mathlib directory:

1
cd mathlib
2
poetry add requests

Poetry updates the pyproject.toml and maintains a lock file (poetry.lock), ensuring reproducibility.

Building and Publishing#

1
poetry build
2
poetry publish --repository testpypi

You can also run poetry publish (without --repository testpypi) to push to the main PyPI.

Conda Packages#

If you work in the data science ecosystem, you might prefer Conda. Conda manages both Python and non-Python libraries, which can be crucial if your application relies on C/C++ libraries.

Pyinstaller for Application Bundling#

Tools like PyInstaller bundle Python applications into standalone executables:

1
pyinstaller --onefile my_script.py

When you distribute the resulting executable, users typically don’t need to install Python or your dependencies separately. This is particularly useful for distributing command-line tools or GUI applications to non-technical users.

Custom Scripts and Entry Points#

In setup.py or pyproject.toml, you can define console scripts that can be run from the command line after installation:

1
setup(
2
    ...
3
    entry_points={
4
        "console_scripts": [
5
            "mathlib-cli=mathlib.main:run_cli",
6
        ],
7
    },
8
)

If someone installs your package, they will have a mathlib-cli command added to their shell, which calls the run_cli() function in mathlib/main.py.

Automatic Versioning#

Tools like setuptools_scm or bumpversion can help auto-increment version numbers based on Git tags, easing the release process.

Handling Native Dependencies#

If your package has native C/C++ extensions, you’ll need to handle compilations. This can be done in setup.py by extending Extension from setuptools. Or you can supply precompiled wheels for different platforms, saving your users from having to compile your extension code themselves.

Local and Manual Deployment#

At the most basic level, you build your package locally and upload it manually to PyPI. This is straightforward but prone to manual steps and errors.

Automated Builds with CI/CD#

Modern development teams often integrate packaging and distribution into Continuous Integration and Continuous Deployment (CI/CD) pipelines. Common platforms include:

• GitHub Actions
• GitLab CI
• Travis CI
• Jenkins

A typical workflow:

1. Commit code to a branch.
2. Run automated tests and linting.
3. On successful tests, build distributions (sdist, wheel).
4. Automatically publish to Test PyPI on merges to a development branch.
5. Optional manual approval to push to production PyPI on merges to main or release tagging.

If using GitHub Actions, for instance, your .github/workflows/publish.yml could look like:

1
name: Publish Python Package
2

3
on:
4
  push:
5
    tags:
6
      - "v*"
7

8
jobs:
9
  build-and-publish:
10
    runs-on: ubuntu-latest
11
    steps:
12
      - uses: actions/checkout@v2
13

14
      - name: Set up Python
15
        uses: actions/setup-python@v2
16
        with:
17
          python-version: 3.9
18

19
      - name: Install dependencies
20
        run: |
21
          pip install --upgrade pip setuptools wheel twine
22

23
      - name: Build
24
        run: |
25
          python setup.py sdist bdist_wheel
26

27
      - name: Publish
28
        run: |
29
          twine upload dist/*
30
        env:
31
          TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
32
          TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}

In this example, packaging occurs automatically whenever you push a tagged release (e.g., v1.0.0) to your repository.

Semantic Versioning at Scale#

When multiple teams and services rely on your packages, consistent versioning is critical:

• Major: Breaking changes.
• Minor: Backward-compatible feature additions.
• Patch: Backward-compatible bug fixes.

Ensuring that each change has a bump in version fosters clarity and trust in your releases.

Multiple Python Version Testing#

Your code might need to support multiple Python versions. Tools like tox let you run your test suite across different Python versions:

1
[tox]
2
envlist = py37, py38, py39, py310
3

4
[testenv]
5
deps = pytest
6
commands = pytest

Code Signing and Security#

Once your package is on PyPI, you might consider cryptographically signing your distributions so users can verify authenticity. Tools like GPG help here, and twine can upload your signed packages.

Private Package Indices#

Organizations often host private PyPI-like repositories (e.g., Nexus, Artifactory) to share internal packages without making them publicly available. This is essential for proprietary code, allowing you to:

• Keep code private.
• Control versioning and distribution within the company.