Module 05 - Packaging and Environments

Reading time: ~10 minutes | Level: Intermediate → Engineering

Before reading further, predict what happens when you run these two commands in sequence on a fresh machine:

pip install project-a   # requires requests==2.28.0
pip install project-b   # requires requests==2.31.0
# Which version of requests is actually installed?
# Which project works correctly?

Show Answer

requests==2.31.0 is installed. project-a silently breaks.

pip is not a package manager in the traditional sense - it is a package installer. It does not track which version of requests was required by project-a. Installing project-b overwrites the requests package with no warning, no error, and no rollback.

If project-a used an API that was removed between 2.28.0 and 2.31.0, it will now raise AttributeError or ImportError at runtime - not at install time. You will debug a production failure that has nothing to do with your code.

This is dependency hell. It is the default state of a machine with a single, shared Python environment. Every project installed into the same Python fights over the same packages.

The solution is isolation: one Python environment per project, with exact version pins, reproduced identically across every developer machine, CI runner, and production container.

This module teaches you the full engineering stack for achieving that.

This is not a hypothetical. Dependency hell has caused production incidents at companies of every size. The tooling Python provides to prevent it - virtual environments, lockfiles, pyproject.toml, and package registries - forms the backbone of professional Python development.

Why Packaging Matters

Packaging solves three distinct engineering problems:

1. Reproducible Environments

Your code works on your laptop. Does it work on your colleague's machine? On the CI runner? On the production server? Without pinned dependencies and isolated environments, the answer is "maybe, for now." Packaging tools make the answer "yes, by design."

2. Dependency Isolation

Different projects need different - sometimes incompatible - versions of the same library. Virtual environments give each project its own isolated copy of Python and its packages, preventing version conflicts entirely.

3. Distribution

You write a useful utility. How does your team install it? How do you publish it to PyPI so the world can use it? Packaging tools provide the standard mechanisms: pyproject.toml describes your package, build creates the distribution artifacts, twine uploads them.

The lockfile is the contract. Every environment resolves to exactly the same packages. Bugs are reproducible. Deployments are predictable.

Module Structure

This module covers six lessons that build on each other in a logical sequence - from the lowest-level concept (what is a virtual environment?) to the highest-level workflow (how do I publish a package to PyPI?).

Lesson	Topic	What You Learn
01	venv and virtualenv	How isolation works at the filesystem level; PATH manipulation; pyenv for Python version management
02	pip and requirements	Dependency resolution; version specifiers; pip-tools for lockfiles; supply-chain security
03	pyproject.toml	The single configuration file replacing setup.py, setup.cfg, MANIFEST.in, and tox.ini
04	Poetry	All-in-one dependency management, virtual environments, and publishing with a clean CLI
05	Semantic Versioning	MAJOR.MINOR.PATCH in practice; pre-release versions; how specifiers like ~= and ^ use semver
06	Publishing Packages	Building sdist and wheel artifacts; TestPyPI and PyPI; CI/CD automated publishing

Module Project

Publish an Internal Utility Package

You will build a small but complete Python library - a collection of data validation utilities - package it with pyproject.toml, manage dependencies with Poetry, version it correctly using semver, and publish it to TestPyPI. A GitHub Actions workflow will automate the publish step on every tagged release.

By the end, you will have gone through the complete lifecycle that every open-source Python package follows: write code → package it → test the package → publish it → install it from the registry.

Prerequisites

Prerequisite	Why It Matters
Module 01 - OOP	Package structure maps to class and module design; __init__.py files control what is public
Module 02 - Functional Programming	Decorators and higher-order functions appear in build tooling and CLI entry points
Module 04 - Testing and Quality	Packages must be tested; pyproject.toml configures pytest, coverage, and linting

How to Use This Module

Each lesson is self-contained but assumes the previous lessons have been read in order. The first two lessons (venv, pip) are foundational - skip them only if you can answer "what does source venv/bin/activate actually modify?" without hesitation.

Lessons 03 and 04 (pyproject.toml and Poetry) overlap intentionally. pyproject.toml is the standard; Poetry is an opinionated tool built on top of it. Learn the standard first.

Lesson 05 (Semantic Versioning) is short but critical - misunderstanding version specifiers is one of the most common sources of subtle dependency bugs.

tip

Read each lesson with a terminal open. Every concept in this module has an immediate, concrete command you can run. The best way to understand what a virtual environment is is to create one, poke around its directory structure, and watch PATH change when you activate it.

The Engineering Standard

Professional Python projects in 2025 share a common structure that this module builds toward:

my-library/
├── src/
│   └── my_library/
│       ├── __init__.py
│       └── core.py
├── tests/
│   ├── conftest.py
│   └── test_core.py
├── pyproject.toml          # single source of truth for metadata, deps, tools
├── poetry.lock             # exact pinned versions for every dependency
├── .python-version         # pyenv: which Python version this project uses
└── .github/
    └── workflows/
        └── publish.yml     # CI/CD: test on push, publish on tag

This structure connects directly to the broader engineering ecosystem:

Docker: Your Dockerfile copies pyproject.toml and the lockfile first, installs dependencies, then copies source. Layer caching means dependency installs are skipped on rebuilds when only source changes.

CI/CD: GitHub Actions, GitLab CI, and CircleCI all have first-class support for pip install and Poetry. The lockfile ensures that the environment in CI matches the environment on every developer's laptop.

PyPI: The Python Package Index hosts over 500,000 packages. Publishing your own package follows the exact same workflow - pyproject.toml → python -m build → twine upload - whether you are publishing to the public PyPI or a private registry like AWS CodeArtifact or Artifactory.

Understanding this stack end-to-end is what separates engineers who write Python from engineers who ship Python.

Key Takeaways

• Dependency hell is the default state of an unmanaged Python installation; packaging tools are how you opt out of it
• Virtual environments provide filesystem-level isolation - one Python, one set of packages, per project
• Lockfiles (requirements.txt pins, poetry.lock) make environments reproducible across machines and time
• pyproject.toml is the modern standard that replaces five legacy configuration files
• The full packaging stack integrates with Docker, CI/CD, and package registries to form a complete delivery pipeline