Packaging Python Projects Like a Pro: Your Step-by-Step Roadmap#

2.1 PyPI (Python Package Index)#

PyPI is Python’s official third-party software repository, akin to an “app store” for Python libraries. It’s where published packages live, making them easily installable using tools like pip.

2.2 pip#

pip is the standard package manager for Python. With pip, users can install or upgrade packages directly from PyPI—or other indexes—and create virtual environments.

2.3 setuptools#

setuptools is one of the most widely used packaging libraries. Historically, it relied on setup.py files for configuration. Today, it also supports integration with pyproject.toml. Many classic Python libraries still use it.

2.4 pyproject.toml#

Introduced via PEP 518, pyproject.toml is a modern configuration file aiding in Python packaging. This central file can specify build dependencies, project metadata, and more, enabling a more standardized approach than older tools.

2.5 Poetry#

Poetry is an alternative packaging and dependency management tool built around pyproject.toml. It aims to simplify project setup, dependency declaration, and packaging through a single tool.

3.1 Package Directory#

• Create a directory named after your package (e.g., my_package).
• This directory should include an init.py file (even if empty) so Python recognizes it as a package.
• Place the core functionality (core.py) within this directory.

3.2 Tests Directory#

• Keep your tests separate in a dedicated tests folder.
• Within tests, create test_*.py files that correspond to the modules being tested.

3.3 Metadata Files#

• README.md: Provide an overview of your project.
• LICENSE: State under what terms your package is distributed and used.
• requirements.txt or pyproject.toml: Declare your build and runtime dependencies.

3.4 Setup Scripts#

• setup.py: Traditional approach for packaging.
• pyproject.toml: Modern approach that can replace or complement setup.py.

4.1 Name#

Specifies the name of your package as it will appear on PyPI and when users install via pip.

4.2 Version#

Indicates your package’s version. Following semantic versioning (e.g., major.minor.patch) is a common best practice.

4.3 Packages#

find_packages() automatically locates sub-packages within your project. You can exclude directories (like tests) to keep them from being installed.

4.4 Dependencies#

The install_requires section lists the libraries that your package needs at runtime. For advanced usage, you might specify version constraints (>=, <=, ==, ~=).

4.5 Optional Arguments#

setup() supports many additional arguments. For instance, entry_points can create command-line scripts from your Python functions, and extras_require can declare optional feature sets.

5.1 requirements.txt#

A simple approach is using a requirements.txt file:

1
numpy==1.21.0
2
requests>=2.25.0

Users can then install these dependencies with:

1
pip install -r requirements.txt

However, this approach doesn’t give granular control over dev vs. prod dependencies, nor does it integrate elegantly with setup.py for packaging.

5.2 setup.py vs. requirements.txt#

setuptools picks up runtime dependencies from install_requires in your setup.py, yet that doesn’t automatically unify them with your requirements.txt. To keep things in sync, some projects either:

• Maintain consistent versions in both files.
• Parse requirements.txt inside setup.py.
• Move to a more modern approach with pyproject.toml.

5.3 pyproject.toml for Dependencies#

Poetry and newer packaging flows keep all necessary info—including dependencies—within pyproject.toml. This ensures that build configurations, version constraints, and dev dependencies are in a single place.

An example snippet from pyproject.toml might look like:

1
[tool.poetry.dependencies]
2
python = "^3.7"
3
numpy = "~1.21.0"
4
requests = ">=2.25.0"
5

6
[tool.poetry.dev-dependencies]
7
pytest = "^6.2"
8
flake8 = "*"

In this format, the caret (^) or tilde (~) illlustrates how versions can be constrained.

7.1 Initializing a Poetry Project#

Installing Poetry:

1
pip install poetry

or follow instructions at https://python-poetry.org/docs/

Next, in your project root:

1
poetry init

Poetry will guide you through steps to create a pyproject.toml.

7.2 Managing Dependencies with Poetry#

You can add new dependencies like so:

1
poetry add requests

For dev dependencies (e.g., for testing):

1
poetry add --dev pytest

Poetry updates pyproject.toml automatically, which might end up looking like:

1
[tool.poetry]
2
name = "my_package"
3
version = "0.3.0"
4
description = "An example package"
5
authors = ["Your Name <your_email@example.com>"]
6
license = "MIT"
7

8
[tool.poetry.dependencies]
9
python = "^3.7"
10
requests = "^2.25"
11
numpy = "^1.21"
12

13
[tool.poetry.dev-dependencies]
14
pytest = "^6.2"

7.3 Locking Dependencies#

One advantage of Poetry is generating a poetry.lock file, which pins exact versions. This ensures consistent installations of your project across different environments.

7.4 Building and Publishing with Poetry#

1
poetry build

This command creates distribution archives. Once happy with your package, you can publish to PyPI:

1
poetry publish

By default, it will ask for your PyPI credentials. If you want to use a test repository first:

1
poetry publish -r testpypi

This ensures you can test your package without affecting the production index.

8.1 Using setuptools#

From the project root with a proper setup.py:

1
python setup.py sdist bdist_wheel

This generates dist/ containing both .tar.gz (sdist) and .whl (wheel) files.

8.2 Checking Your Distribution#

After creating your distributions, test installing them in a clean virtual environment:

1
pip install dist/my_package-0.2.1-py3-none-any.whl

This assures that everything is packaged correctly.

10.1 TestPyPI#

TestPyPI is a staging environment to test package uploads. First create an account on both test.pypi.org and pypi.org. Then:

1. Update your credentials in ~/.pypirc:

   1
   [distutils]
   2
   index-servers =
   3
       pypi
   4
       testpypi
   5
   
   6
   [testpypi]
   7
   repository: https://test.pypi.org/legacy/
   8
   username: <your_username>
   9
   password: <your_password>
   10
   
   11
   [pypi]
   12
   repository: https://upload.pypi.org/legacy/
   13
   username: <your_username>
   14
   password: <your_password>
   

2. Upload to TestPyPI:

   1
   twine upload --repository testpypi dist/*
   

3. Verify installation:

   1
   pip install --index-url https://test.pypi.org/simple/ my_package
   

10.2 PyPI#

Once satisfied, upload to PyPI:

1
twine upload dist/*

Or let Poetry handle it:

1
poetry publish

Now anyone can pip install your_package directly from PyPI.

11.1 Semantic Versioning#

Semantic versioning is a convention that expresses your project’s stability and the significance of changes in each release:

• Major (e.g., 2.x.x): Breaking changes.
• Minor (e.g., 1.1.x): Backwards-compatible new features.
• Patch (e.g., 1.0.1): Backwards-compatible bug fixes only.

11.2 Continuous Integration and Deployment#

Setting up CI/CD helps automate testing and publishing:

• GitHub Actions or GitLab CI can trigger tests upon push or pull requests.
• Condition publishing steps on passing tests.
• Tagging a release in Git can automatically trigger a build pipeline that publishes to PyPI.

11.3 Distribution across Python Versions#

Use python_requires in your setup.py or specify multiple classifiers in pyproject.toml to ensure your package only installs on compatible Python versions:

1
python_requires=">=3.6"

That line will prevent accidental installs on older Python versions.

11.4 Multi-Platform Wheels#

If your package uses compiled extensions (C/C++ code), you might consider building multiple platform-specific wheels. Tools such as cibuildwheel can automate cross-platform wheel building for Linux, macOS, and Windows.

11.5 Supporting Additional Entry Points#

entry_points in setuptools, or scripts in Poetry, allow you to define console scripts. For example, a user can type my_tool at the command line to automatically run a function in your package. In setup.py:

1
setup(
2
    ...,
3
    entry_points={
4
        "console_scripts": [
5
            "my_tool=my_package.core:main",
6
        ]
7
    },
8
)

Now, if you define a main() function in my_package/core.py, users can execute your tool directly once the package is installed.

11.6 Including Data Files#

If your library needs data files (like CSVs, JSONs, or other resources), you must ensure they’re properly included at installation time. Consider using MANIFEST.in or configuring setup() with include_package_data=True:

MANIFEST.in example:

1
include my_package/data/*.json
2
include my_package/data/*.csv

This ensures these files are bundled within the source distribution.