pip and requirements - Dependency Management in Practice

Reading time: ~35 minutes | Level: Intermediate → Engineering

Before reading further, diagnose why this application breaks six months after it was first deployed:

# requirements.txt written in January
requests
flask
sqlalchemy

# Six months later, on a new machine:
pip install -r requirements.txt
# App starts. Then crashes with:
# AttributeError: 'HTTPResponse' object has no attribute 'auto_close'

Show Answer

In January, pip install requests installed requests==2.28.2 - the latest version at the time. Your code used an internal behavior of HTTPResponse that worked in 2.28.2.

Six months later, pip install requests installs requests==2.31.0. The auto_close attribute was removed in an intermediate release. Your requirements.txt never specified a version, so pip happily installs the latest, breaking your code silently at runtime.

The fix: pin your versions. But pinning by hand does not scale - you need a tool that pins both your direct dependencies and their transitive dependencies.

# requirements.txt that would not have broken
requests==2.28.2
flask==2.3.3
sqlalchemy==2.0.21
certifi==2023.7.22       # transitive dep of requests
charset-normalizer==3.2.0
idna==3.4
urllib3==2.0.4

Every version pinned, including every package that requests, flask, and sqlalchemy themselves depend on. This file is a lockfile - a complete, reproducible specification of the environment. This is what pip-tools generates automatically.

Unpinned requirements.txt files are a time bomb. Understanding why requires knowing how pip resolves dependencies, what version specifiers actually mean, and what tools exist to make pinning automatic and maintainable.

What You Will Learn

• How pip's dependency resolver works (and why it was rewritten in pip 20.3)
• Version specifiers: ==, >=, <=, ~=, != - what each means in practice
• pip install flags that matter in production: --upgrade, --no-deps, --dry-run
• pip list, pip show, pip check - inspecting your environment
• pip freeze vs pip list --format=freeze - the difference and when it matters
• requirements.txt best practices: apps vs libraries
• The two-file pattern: requirements.in + requirements.txt
• pip-tools: pip-compile and pip-sync for automated lockfile management
• Layered requirements for dev/prod separation
• Editable installs (pip install -e .)
• Hash verification and pip audit for supply-chain security
• Private package indexes

Prerequisites

• Lesson 01 - venv and virtualenv (always install into an activated virtual environment)
• Understanding of basic terminal usage

Part 1 - How pip Resolves Dependencies

When you run pip install flask, pip does not just download Flask. It reads Flask's metadata to find what Flask depends on - Werkzeug, Jinja2, MarkupSafe, click, itsdangerous - and then resolves their dependencies, recursively, until it has a complete set of packages with no conflicts.

The Dependency Resolver

Before pip 20.3 (released November 2020), pip used a greedy resolver: it installed packages in order, and if a conflict was discovered late, it would fail or install an incompatible version without warning. This was the source of many silent breakages.

pip 20.3 introduced a backtracking resolver: when it encounters a conflict, it backtracks and tries alternative versions. This is correct behavior but can be slow with many packages.

# Watch the resolver work:
pip install flask --verbose 2>&1 | grep -E "(Collecting|Downloading|Installing)"

# Collecting flask
# Collecting Werkzeug>=3.0.0 (from flask)
# Collecting Jinja2>=3.1.2 (from flask)
# Collecting click>=8.1.3 (from flask)
# Collecting itsdangerous>=2.1.2 (from flask)
# Collecting blinker>=1.6.2 (from flask)
# Collecting MarkupSafe>=2.0 (from Jinja2->flask)
# Downloading Flask-3.0.0-py3-none-any.whl
# ...

Every package in that output ends up in your site-packages. None of them are in your requirements.txt. When you update Flask, its dependencies update too - invisibly, unless you pin them.

Part 2 - Version Specifiers

pip uses PEP 440 version specifiers. Understanding them exactly prevents subtle bugs.

The Specifiers

requests==2.28.0       # Exact match: ONLY 2.28.0
requests>=2.28.0       # Greater than or equal: 2.28.0, 2.29.0, 3.0.0, ...
requests<=2.28.0       # Less than or equal: 2.28.0, 2.27.0, ...
requests>2.28.0        # Strictly greater: 2.28.1, 2.29.0, 3.0.0, ...
requests<2.29.0        # Strictly less: 2.28.x and below
requests!=2.28.1       # Exclude: anything except exactly 2.28.1
requests>=2.28,<3      # Range: any 2.x.x from 2.28 onwards
requests~=2.28.0       # Compatible release (see below)
requests~=2.28         # Compatible release (see below)

The Compatible Release Operator (~=)

This is the most commonly misunderstood specifier:

~=2.28.0   means   >=2.28.0, ==2.28.*   (patch-level compatible: 2.28.0, 2.28.1, 2.28.9, but NOT 2.29.0)
~=2.28     means   >=2.28, ==2.*        (minor-level compatible: 2.28, 2.29, 2.30, but NOT 3.0)

The rule: ~=X.Y.Z means "at least X.Y.Z, but stay within the X.Y.* series." ~=X.Y means "at least X.Y, but stay within the X.* series."

warning

~=2.28 means >=2.28, <3 - it allows 2.29, 2.30, and any later 2.x release. It does NOT mean "only patch updates from 2.28." If you want only patch updates, use ~=2.28.0. This confusion causes real incidents when a breaking minor release lands.

# Test what version pip would install without actually installing:
pip install "requests~=2.28.0" --dry-run

Specifiers for Libraries vs Applications

Context	Recommendation	Reason
Application (requirements.txt)	requests==2.28.2 (exact pin)	Reproducibility; prevent unexpected upgrades
Library (pyproject.toml deps)	requests>=2.28,<3 (range)	Allow users to upgrade; avoid conflicts with their other packages

A library with requests==2.28.2 in its dependencies forces every user of your library to use exactly that version, even if their other packages need a newer one. Use ranges in libraries.

Part 3 - Essential pip Commands

Installing

pip install flask                        # latest version
pip install "flask==3.0.0"              # exact version
pip install "flask>=2.3,<4"             # range
pip install flask werkzeug click         # multiple packages
pip install -r requirements.txt          # from file
pip install -r requirements.txt --upgrade  # upgrade to latest matching specs
pip install --no-deps flask              # install flask only, skip deps (dangerous)
pip install flask --dry-run              # show what would be installed, do nothing

Inspecting Your Environment

pip list                                # all installed packages
pip list --outdated                     # packages with newer versions available
pip show flask                          # metadata for one package
pip check                               # find broken/conflicting dependencies

pip check is underused and critically important:

pip check
# No broken requirements found.

# Or:
# flask 3.0.0 requires werkzeug>=3.0.0, but you have werkzeug 2.3.7 which is incompatible.

Run pip check after any install or upgrade. A broken environment fails in mysterious ways at runtime.

Generating Requirements

pip freeze                              # all installed packages, pinned, pip-install format
pip freeze > requirements.txt           # save to file
pip list --format=freeze                # same output as pip freeze (same thing)

The difference between pip freeze and pip list:

pip list
# Package    Version
# --------- -------
# flask     3.0.0
# pip       23.3.1
# setuptools 68.2.2
# werkzeug  3.0.1

pip freeze
# flask==3.0.0
# werkzeug==3.0.1
# (pip and setuptools are NOT included - pip omits its own metadata and build tools)

pip freeze excludes pip, setuptools, and wheel from its output. For everything else, both commands show the same pinned versions.

tip

Do not use pip freeze > requirements.txt as your primary workflow. It captures everything in the environment - including packages you installed manually for debugging, packages that are transitive deps of dev tools, and packages you meant to add to requirements.txt but forgot. Use pip-tools instead (see Part 4).

Part 4 - pip-tools: The Right Way to Manage Lockfiles

pip-tools provides two commands that separate the human-readable dependency specification from the machine-generated lockfile:

• pip-compile: takes a requirements.in (abstract, human-written) and generates a fully pinned requirements.txt (lockfile)
• pip-sync: installs exactly the packages in requirements.txt and removes everything else - making the venv match the spec precisely

Installation

pip install pip-tools

The Two-File Pattern

requirements.in - what you write:

# requirements.in
flask>=3.0
sqlalchemy>=2.0
requests~=2.31.0

Only your direct dependencies, with loose constraints. This is the file humans edit.

requirements.txt - what pip-compile generates:

pip-compile requirements.in

# requirements.txt (generated by pip-compile, DO NOT EDIT)
# This file is generated from requirements.in by pip-compile.
# To update, run: pip-compile requirements.in
#
blinker==1.7.0
    # via flask
certifi==2023.11.17
    # via requests
charset-normalizer==3.3.2
    # via requests
click==8.1.7
    # via flask
flask==3.0.0
    # via -r requirements.in
greenlet==3.0.1
    # via sqlalchemy
idna==3.6
    # via requests
itsdangerous==2.1.2
    # via flask
jinja2==3.1.2
    # via flask
markupsafe==2.1.3
    # via
    #   jinja2
    #   werkzeug
requests==2.31.0
    # via -r requirements.in
sqlalchemy==2.0.23
    # via -r requirements.in
typing-extensions==4.8.0
    # via sqlalchemy
urllib3==2.1.0
    # via requests
werkzeug==3.0.1
    # via flask

Every transitive dependency is pinned. Comments show which direct dependency pulled each package in. This file is your lockfile - commit it.

pip-sync vs pip install

# pip install -r requirements.txt
# ADDS packages from the file. Does not remove packages not in the file.
# If you manually installed a package for debugging, it stays.

# pip-sync requirements.txt
# Makes the environment EXACTLY match the file.
# Removes packages not listed. Installs missing ones. Upgrades/downgrades as needed.

Use pip-sync in CI and for reproducible setups. It guarantees the environment matches the lockfile exactly.

Upgrading Dependencies

# Upgrade all packages to latest versions matching requirements.in constraints:
pip-compile --upgrade requirements.in

# Upgrade just one package:
pip-compile --upgrade-package flask requirements.in

# Then sync the environment:
pip-sync requirements.txt

This workflow makes upgrades explicit and traceable: the diff in requirements.txt shows exactly which packages changed and to what versions.

Part 5 - Layered Requirements

Production projects typically have different dependency sets for different contexts:

requirements/
├── base.txt         # runtime dependencies (generated from base.in)
├── dev.txt          # base + dev tools (generated from dev.in)
└── prod.txt         # base + prod-only tools (generated from prod.in)

requirements/base.in:

flask>=3.0
sqlalchemy>=2.0
requests~=2.31.0

requirements/dev.in:

-c base.txt          # constrained by base.txt - use same versions of shared packages
pytest>=7.4
pytest-cov>=4.1
ruff>=0.1
mypy>=1.6

requirements/prod.in:

-c base.txt
gunicorn>=21.2
sentry-sdk>=1.35

# Compile all three:
pip-compile requirements/base.in -o requirements/base.txt
pip-compile requirements/dev.in -o requirements/dev.txt
pip-compile requirements/prod.in -o requirements/prod.txt

# Use in different contexts:
pip-sync requirements/dev.txt    # local development
pip-sync requirements/prod.txt   # production deployment

Part 6 - Editable Installs

When developing a package locally, you want changes to your source code to be reflected immediately without reinstalling. pip install -e . does this:

# Inside your project directory (where pyproject.toml lives):
pip install -e .

# Or with optional dependencies:
pip install -e ".[dev]"

How It Works

With modern pip (21.3+) and pyproject.toml, editable installs create a .pth file or a direct path reference in site-packages:

ls venv/lib/python3.11/site-packages/ | grep -E "(egg-link|pth|my_package)"
# my-package.pth
# OR
# __editable__.my_package-0.1.0.pth

That file contains the path to your source directory. When Python imports my_package, it finds the .pth file, reads the path, and imports directly from your source tree. Edit src/my_package/core.py and the next import my_package picks up the change - no reinstall needed.

# Verify editable install:
pip show my-package
# Location: /path/to/project/src   ← points to your source, not site-packages

note

pip's dependency resolver can be slow when resolving environments with many packages. If pip install seems stuck, it is usually backtracking through version combinations. You can watch it with pip install flask -v (verbose). For very large environments, pip-tools with a cache is significantly faster than bare pip for repeated installs.

Part 7 - Hash Verification and Supply-Chain Security

PyPI packages can be modified after you first install them (though PyPI itself is hardened against this). For maximum security - especially in regulated industries or sensitive infrastructure - verify cryptographic hashes:

# Generate requirements.txt with hashes:
pip-compile --generate-hashes requirements.in

# The output includes SHA-256 hashes:
# flask==3.0.0 \
#     --hash=sha256:21128f47e4e3b9d597a3e8521a329bf56909d690ffc7f5e1a2ae... \
#     --hash=sha256:c3...

Installing with hash verification:

pip install --require-hashes -r requirements.txt

If any package's downloaded content does not match its hash, the install fails. This prevents supply-chain attacks where a malicious actor substitutes a package between your pip-compile and another developer's pip install.

pip audit

pip audit scans your installed packages against the Python Packaging Advisory Database and OSV for known CVEs:

pip install pip-audit
pip audit

# Example output:
# Found 1 known vulnerability in 1 package
# Name     Version ID                  Fix Versions
# -------- ------- ------------------- ------------
# requests 2.26.0  GHSA-j8r2-6x86-q33q 2.27.1

Run pip audit in CI on every pull request. A dependency with a critical CVE that slips into a release is a production incident waiting to happen.

danger

pip install --upgrade without a lockfile on a production server is one of the most dangerous operations in Python deployment. It upgrades every installed package to its latest version in one command, with no record of what changed, no rollback capability, and no testing. Use pip-sync requirements.txt with a tested, committed lockfile instead.

Part 8 - Private Package Indexes

Teams often publish internal packages to private registries. pip supports multiple package indexes:

# Replace PyPI entirely (for air-gapped environments):
pip install mypackage --index-url https://my.registry.example.com/simple/

# Add a private index in addition to PyPI:
pip install mypackage --extra-index-url https://my.registry.example.com/simple/

In requirements.txt or pip.conf:

# pip.conf (Linux: ~/.config/pip/pip.conf, macOS: ~/Library/Application Support/pip/pip.conf)
[global]
extra-index-url = https://my.registry.example.com/simple/

Authentication

For private registries requiring authentication, use ~/.netrc:

# ~/.netrc
machine my.registry.example.com
  login myuser
  password mytoken

Or in the URL (less secure - visible in logs):

pip install mypackage --extra-index-url https://myuser:[email protected]/simple/

For CI, use environment variables via the PIP_EXTRA_INDEX_URL environment variable:

export PIP_EXTRA_INDEX_URL="https://__token__:${REGISTRY_TOKEN}@my.registry.example.com/simple/"
pip install mypackage

Common private registries: AWS CodeArtifact, Google Artifact Registry, Azure Artifacts, GitLab Package Registry, JFrog Artifactory.

Part 9 - Graded Practice

Level 1 - Predict and Identify

Question 1: What does pip check do, and when should you run it?

Show Answer

pip check verifies that all installed packages have their dependencies satisfied - no missing packages, no version conflicts. It does not install or change anything; it only reports.

Run it:

• After pip install -r requirements.txt to verify the install was clean
• After manually installing any package
• In CI, as a health check step before running tests
• When debugging mysterious ImportError or AttributeError failures that seem unrelated to your code changes

Question 2: You have requests~=2.31.0 in your requirements.in. Which of these versions would pip-compile accept as satisfying this constraint?

a) requests==2.31.5
b) requests==2.32.0
c) requests==3.0.0
d) requests==2.30.9

Show Answer

Only (a) requests==2.31.5 satisfies ~=2.31.0.

~=2.31.0 means >=2.31.0, ==2.31.* - stay within the 2.31.* series.

• 2.31.5: within 2.31.* - YES
• 2.32.0: a new minor version - NO
• 3.0.0: a new major version - NO
• 2.30.9: below 2.31.0 - NO

If you wanted to allow 2.32.0, you would use ~=2.31 (equivalent to >=2.31, ==2.*).

Question 3: What is the difference between pip install -r requirements.txt and pip-sync requirements.txt?

Show Answer

pip install -r requirements.txt adds packages from the file to the current environment. It does not remove packages that are present in the environment but not listed in the file. The environment can accumulate extra packages over time.

pip-sync requirements.txt makes the environment exactly match the file. It:

• Installs packages listed in the file that are not yet installed
• Upgrades or downgrades packages whose installed version differs from the pinned version
• Removes packages that are present in the environment but not listed in the file

pip-sync gives you a guarantee: after it runs, the environment contains exactly and only the packages in requirements.txt. Use pip-sync when reproducibility is required (CI, onboarding, deployments). Use pip install -r when you want to add packages without disturbing the rest of the environment.

Level 2 - Debug and Fix

Scenario: Your CI pipeline has been running pip install -r requirements.txt for six months. Today it fails with:

ERROR: pip's dependency resolver does not currently take into account all the packages
that are installed. This behaviour is the source of the following dependency conflicts.
flask 3.0.0 requires werkzeug>=3.0.0, but you have werkzeug 2.3.7 which is incompatible.

Your requirements.txt contains:

flask==3.0.0
werkzeug==2.3.7

How did this happen, and what is the correct fix?

Show Answer

How it happened: When this requirements.txt was written (likely with pip freeze), Flask 3.0.0 had not yet been released - or a version of Flask that worked with Werkzeug 2.3.7 was current. Over time, someone upgraded Flask to 3.0.0 without regenerating the full lockfile. Flask 3.0.0 requires werkzeug>=3.0.0, but the lockfile still pins werkzeug==2.3.7.

pip installed flask==3.0.0 and werkzeug==2.3.7 as specified, then detected the conflict. In newer pip versions this is an error; in older versions it would silently install the incompatible combination and fail at runtime.

The fix:

1. Switch to pip-tools to prevent this class of problem:

pip install pip-tools

# Create requirements.in with just your direct deps:
cat > requirements.in << 'EOF'
flask>=3.0
EOF

# Regenerate the full lockfile:
pip-compile requirements.in

# The generated requirements.txt will have compatible versions:
# flask==3.0.0
# werkzeug==3.0.1   ← compatible version selected automatically
# ...

2. Sync your environment:

pip-sync requirements.txt
pip check   # verify no conflicts

Root cause: pip freeze does not understand dependency relationships - it just lists installed versions. If you manually edit or upgrade one package, the freeze output can become internally inconsistent. pip-compile solves for a consistent set automatically.

Level 3 - Design

Challenge: You are setting up the dependency management workflow for a new FastAPI service. The service has:

• Runtime dependencies: fastapi, sqlalchemy, pydantic, httpx
• Dev-only tools: pytest, pytest-asyncio, ruff, mypy, coverage
• Production-only: uvicorn[standard], sentry-sdk

Design the complete requirements/ structure using pip-tools, including the .in files, how to compile them, how they reference each other, and the CI commands for install and audit.

Show Answer

Directory structure:

myservice/
├── requirements/
│   ├── base.in
│   ├── base.txt          # generated, committed
│   ├── dev.in
│   ├── dev.txt           # generated, committed
│   ├── prod.in
│   └── prod.txt          # generated, committed
└── Makefile              # convenience targets

requirements/base.in:

# Runtime dependencies - used in all environments
fastapi>=0.104
sqlalchemy>=2.0
pydantic>=2.4
httpx>=0.25

requirements/dev.in:

# Development and testing tools
# -c constrains shared packages to versions resolved in base.txt
-c base.txt

pytest>=7.4
pytest-asyncio>=0.21
ruff>=0.1
mypy>=1.6
coverage[toml]>=7.3

requirements/prod.in:

# Production server + observability
-c base.txt

uvicorn[standard]>=0.24
sentry-sdk[fastapi]>=1.35

Compilation:

# Compile in dependency order (base first, then layered)
pip-compile requirements/base.in -o requirements/base.txt
pip-compile requirements/dev.in -o requirements/dev.txt --generate-hashes
pip-compile requirements/prod.in -o requirements/prod.txt --generate-hashes

Makefile:

.PHONY: compile sync-dev sync-prod audit upgrade

compile:
	pip-compile requirements/base.in -o requirements/base.txt
	pip-compile requirements/dev.in -o requirements/dev.txt --generate-hashes
	pip-compile requirements/prod.in -o requirements/prod.txt --generate-hashes

sync-dev:
	pip-sync requirements/dev.txt

sync-prod:
	pip-sync requirements/prod.txt

audit:
	pip audit -r requirements/base.txt
	pip audit -r requirements/prod.txt

upgrade:
	pip-compile --upgrade requirements/base.in -o requirements/base.txt
	pip-compile --upgrade requirements/dev.in -o requirements/dev.txt
	pip-compile --upgrade requirements/prod.in -o requirements/prod.txt

CI pipeline (GitHub Actions excerpt):

- name: Install dependencies
  run: |
    python -m pip install pip-tools
    pip-sync requirements/dev.txt

- name: Security audit
  run: pip audit -r requirements/base.txt

- name: Check for conflicts
  run: pip check

- name: Run tests
  run: pytest

Key design decisions:

1. -c base.txt in dev and prod .in files ensures shared packages stay consistent across environments
2. --generate-hashes on prod and dev lockfiles enables supply-chain verification
3. The Makefile upgrade target makes version upgrades explicit and traceable via git diff
4. pip audit in CI catches CVEs before they reach production

Key Takeaways

• pip installs packages but does not track which project needs which version; use virtual environments per project to prevent conflicts
• Unpinned requirements.txt files install whatever is latest at install time - they are not lockfiles
• ~=2.28.0 pins to the 2.28.* series; ~=2.28 allows any 2.* - know the difference before using compatible-release
• pip check finds broken dependency relationships in your current environment; run it after every install
• pip-tools separates human intent (requirements.in) from machine-generated lockfiles (requirements.txt); use it instead of pip freeze
• pip-sync makes your environment match the lockfile exactly, including removing extra packages; prefer it over pip install -r for reproducibility
• Layer your requirements: base.txt for runtime, dev.txt for development, prod.txt for production
• pip audit scans for CVEs; --require-hashes verifies package integrity - use both in security-sensitive deployments
• Never pip install --upgrade on a production server without a tested lockfile