venv and virtualenv - Python Environment Isolation

Reading time: ~30 minutes | Level: Intermediate → Engineering

Before reading further, trace exactly what happens when you run these commands on macOS or Linux:

which python        # /usr/bin/python3
pip install flask
which python        # still /usr/bin/python3
python -c "import flask; print(flask.__file__)"
# Where does this file live?
# What happens when you upgrade your OS and it ships a new Python?
# What happens when another project needs a different version of Flask?

Show Answer

On macOS, flask.__file__ will print something like:

/Library/Python/3.11/lib/python/site-packages/flask/__init__.py

or on Linux:

/usr/local/lib/python3.11/site-packages/flask/__init__.py

This is the system Python's site-packages directory - the single location where every Python package installed with pip on this machine lives. All your projects share it.

When you upgrade macOS, the system Python is replaced. The new Python has an empty site-packages. All your installed packages disappear. You have no record of what was installed or at what versions.

When another project needs flask==2.0.0 and your current project needs flask==3.0.0, one of them will be broken. The last pip install flask==X.Y.Z wins.

The solution is a virtual environment: a self-contained directory with its own Python interpreter and its own site-packages, completely separate from the system Python and from every other project's environment.

Installing into system Python is asking for trouble. Understanding why requires understanding what a virtual environment actually is - not just how to create one, but what the directory structure looks like and what activate actually does to your shell.

What You Will Learn

What a virtual environment is: the filesystem structure that makes isolation possible
How source venv/bin/activate works: PATH prepending and nothing else
pyvenv.cfg: the file that defines a venv and links it to its parent Python
python -m venv: all flags worth knowing for production use
virtualenv: the third-party alternative and when to prefer it
How pip install finds the right site-packages using sys.prefix
pyenv: managing multiple Python versions on one machine
When and why you need a venv inside a Docker container

Prerequisites

Module 05 Lesson 00 - Overview (understanding why isolation matters)
Basic terminal/shell usage (cd, ls, environment variables)

Part 1 - The Filesystem Structure of a Virtual Environment

A virtual environment is not magic. It is a directory with a specific layout that Python knows how to interpret.

Run this and look at what was created:

python3 -m venv myenv
find myenv -maxdepth 4 -type f | sort | head -40

The output reveals a structure like this:

myenv/
├── bin/                          # (Scripts/ on Windows)
│   ├── activate                  # bash/zsh activation script
│   ├── activate.csh              # csh activation script
│   ├── activate.fish             # fish activation script
│   ├── python -> python3         # symlink to the Python interpreter
│   ├── python3 -> /usr/bin/python3   # symlink to system Python
│   └── pip                       # pip for THIS environment only
├── include/
│   └── python3.11/               # C headers for extension modules
├── lib/
│   └── python3.11/
│       └── site-packages/        # packages installed in THIS environment
│           ├── pip/
│           └── setuptools/
└── pyvenv.cfg                    # the file that makes this directory a venv

The key insight: the virtual environment does not contain a separate Python interpreter binary (unless you use --copies). It contains a symlink to the system Python. What it does contain is a separate site-packages directory and a pyvenv.cfg file that redirects Python's package search path to that directory instead of the system one.

pyvenv.cfg - The Configuration File That Makes a venv a venv

cat myenv/pyvenv.cfg

home = /usr/bin
include-system-site-packages = false
version = 3.11.6

Key	Meaning
`home`	The directory of the Python interpreter this venv was created from
`include-system-site-packages`	If `true`, venv packages are added to system packages; almost always `false`
`version`	The Python version string - informational

When Python starts and finds a pyvenv.cfg file in a parent directory, it reads it and sets sys.prefix and sys.exec_prefix to the venv directory instead of the system Python directory. This is what redirects all package lookups to myenv/lib/python3.11/site-packages/ instead of the system site-packages.

Part 2 - How Activation Works

This is the most common misconception about virtual environments: source venv/bin/activate does something sophisticated. It does not.

# Look at what activate actually does:
cat myenv/bin/activate | head -30

The critical lines:

VIRTUAL_ENV="/path/to/myenv"
export VIRTUAL_ENV

_OLD_VIRTUAL_PATH="$PATH"
PATH="$VIRTUAL_ENV/bin:$PATH"
export PATH

That is all. Activation prepends myenv/bin/ to your PATH. When you now type python, the shell finds myenv/bin/python (the symlink) before it finds /usr/bin/python3. When you type pip, it finds myenv/bin/pip.

It also sets $VIRTUAL_ENV and modifies your shell prompt to show the venv name - cosmetic changes for your convenience.

deactivate

deactivate is a shell function defined by the activate script. It restores $PATH from $_OLD_VIRTUAL_PATH and unsets $VIRTUAL_ENV. Nothing more.

tip

You can check whether you are inside a virtual environment from Python code:

import sys
import os

# Method 1: check sys.prefix vs sys.base_prefix
in_venv = sys.prefix != sys.base_prefix
print(f"In virtual environment: {in_venv}")

# Method 2: check VIRTUAL_ENV environment variable
venv_path = os.environ.get("VIRTUAL_ENV")
print(f"Virtual environment path: {venv_path}")

Part 3 \text{---} python -m venv in Production

python -m venv is a stdlib module available in every Python 3.3+ installation. It is the right tool for creating virtual environments in production workflows.

Basic Usage

# Create a virtual environment named venv (conventional name)
python3 -m venv venv

# Activate it
source venv/bin/activate        # bash/zsh
source venv/bin/activate.fish   # fish
venv\Scripts\activate           # Windows PowerShell

# Verify
which python
python --version
pip --version

Useful Flags

# Custom shell prompt instead of the directory name
python3 -m venv venv --prompt myproject

# Use copies instead of symlinks (useful on network filesystems)
python3 -m venv venv --copies

# Skip installing pip (use this in Docker to keep image size minimal)
python3 -m venv venv --without-pip

# Upgrade pip and setuptools to latest after creation
python3 -m venv venv --upgrade-deps

# Allow access to system site-packages (rarely correct)
python3 -m venv venv --system-site-packages

warning

python -m venv vs python3 -m venv \text{---} on some Linux distributions, python refers to Python 2 and python3 refers to Python 3. On macOS with Homebrew or pyenv, python is usually Python 3. Always verify with python --version and python3 --version before creating a venv. Use python3 -m venv to be explicit on systems where ambiguity exists.

How pip install Finds the Right site-packages

When you run pip install flask inside an activated venv, pip uses sys.prefix to locate the target site-packages:

import sys
print(sys.prefix)         # /path/to/myenv
print(sys.exec_prefix)    # /path/to/myenv
print(sys.base_prefix)    # /usr  (the system Python, unchanged)

pip installs to {sys.prefix}/lib/pythonX.Y/site-packages/. Because sys.prefix points to the venv directory (set by pyvenv.cfg at interpreter startup), the package lands in the venv, not the system.

Part 4 \text{---} virtualenv (Third-Party Alternative)

virtualenv is the original virtual environment tool, now a third-party package. It predates python -m venv and has more features:

pip install virtualenv

# Create environments for different Python versions
virtualenv venv --python=python3.11
virtualenv venv --python=python3.12

# Faster creation (uses seed packages from cache)
virtualenv venv

# Built-in activators for fish, csh, PowerShell, xonsh, nushell
ls venv/bin/activate*

Feature	`python -m venv`	`virtualenv`
Requires installation	No (stdlib)	Yes (`pip install virtualenv`)
Speed	Slower (no cache)	Faster (uses seed package cache)
Python version selection	Must use that Python binary	`--python=python3.X` flag
Activator scripts	bash, csh, fish	bash, csh, fish, PowerShell, xonsh, nushell
Seeder plugins	No	Yes (supports pip, setuptools, wheel seeds)

For most projects, python -m venv is sufficient. Use virtualenv when you need faster environment creation in CI (the package seed cache is significant at scale) or need PowerShell activators.

Part 5 \text{---} pyenv: Managing Python Versions

A virtual environment isolates packages. It does not help you run multiple Python versions (3.10, 3.11, 3.12) on the same machine. That is pyenv's job.

Installation

# macOS
brew install pyenv

# Linux
curl https://pyenv.run | bash
# Then add to ~/.bashrc or ~/.zshrc:
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"

Core Commands

# List all available Python versions
pyenv install --list | grep "3.12"

# Install a specific version
pyenv install 3.12.0
pyenv install 3.11.6

# Set the global default
pyenv global 3.12.0

# Set a version for just this project directory (creates .python-version)
pyenv local 3.11.6

# Check which Python is active and why
pyenv version
# 3.11.6 (set by /path/to/project/.python-version)

# List installed versions
pyenv versions

.python-version

When you run pyenv local 3.11.6, pyenv writes a .python-version file:

cat .python-version
# 3.11.6

Any directory containing this file (and its subdirectories) uses that Python version automatically. Commit this file to your repository \text{---} it tells other developers which Python version this project targets.

The Complete Project Setup Workflow

pyenv-virtualenv Plugin

# Install the plugin
brew install pyenv-virtualenv   # macOS
# or: git clone https://github.com/pyenv/pyenv-virtualenv $(pyenv root)/plugins/pyenv-virtualenv

# Create a named virtualenv attached to a specific Python version
pyenv virtualenv 3.11.6 myproject-env

# Activate it
pyenv activate myproject-env

# Set it as the local env for a directory (combines pyenv local + activate)
pyenv local myproject-env

This is the approach used in teams where multiple projects on the same machine need different Python versions and different packages.

note

conda environments are a completely different system. conda manages non-Python packages (C libraries, CUDA, R) in addition to Python packages. It has its own package format, its own registry (conda-forge, Anaconda), and its own solver. conda activate myenv works similarly to source venv/bin/activate, but the underlying mechanism is different. Do not mix conda and pip environments without understanding the implications - pip install inside a conda environment can corrupt it.

Part 6 - Docker Containers Are Not Virtual Environments

A Docker container provides OS-level isolation: its own filesystem, its own process tree, its own network namespace. It is not a virtual environment.

The question arises: if a Docker container has its own Python, why do you need a virtual environment inside it?

In most production containers, you do not need one. The container itself provides the isolation. Install packages directly with pip install and they go to the container's Python.

But there are two important exceptions:

Multi-stage builds: In a Docker multi-stage build, you build in one stage and copy to a minimal runtime stage. If you create a venv in the build stage, you can copy the entire venv directory to the runtime stage with a single COPY command, without worrying about which system paths pip used.

# Build stage
FROM python:3.12-slim AS builder
WORKDIR /app
RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"
COPY requirements.txt .
RUN pip install -r requirements.txt

# Runtime stage
FROM python:3.12-slim
COPY --from=builder /opt/venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"
COPY src/ ./src/
CMD ["python", "-m", "myapp"]

Development containers: When you mount your local source directory into a container for development (docker run -v $(pwd):/app), you want packages installed in the venv on the mounted volume to persist between container restarts, without being mixed with the container's system packages.

danger

Never run sudo pip install on any machine. On Linux, the system Python (/usr/bin/python3) is used by system tools \text{---} package managers, update scripts, apt hooks. Installing packages with sudo pip install can overwrite library versions that system tools depend on, causing OS-level failures that require reinstalling the operating system to fix. This has happened. Use virtual environments instead.

Part 7 \text{---} Graded Practice

Level 1 \text{---} Predict and Identify

Question 1: What is the only thing that source venv/bin/activate modifies in your shell environment (besides the prompt)?

Show Answer

It modifies the PATH environment variable, prepending venv/bin/ to it. That is all. No environment variables related to Python's module search path (PYTHONPATH) are changed. The isolation comes entirely from PATH determining which python binary is found first.

Question 2: You create a venv with python3 -m venv myenv. You then upgrade the system Python from 3.11 to 3.12. Does the venv still work?

Show Answer

It depends on whether you used symlinks (default) or copies (--copies).

With symlinks (default): myenv/bin/python is a symlink to /usr/bin/python3. After the upgrade, /usr/bin/python3 now points to Python 3.12. The symlink still resolves, but Python 3.12 will not find packages installed for 3.11 in myenv/lib/python3.11/site-packages/. The venv is broken in a subtle way \text{---} Python starts but packages fail to import.

With --copies: the Python binary is a copy. It is still 3.11 even after the system upgrade. The venv continues to work until the 3.11 binary itself is removed.

Best practice: treat a venv as disposable. Delete it and recreate it after a Python version upgrade.

Question 3: What does pyvenv.cfg contain, and which program reads it?

Show Answer

pyvenv.cfg contains at minimum:

home: the directory of the Python interpreter the venv was created from
include-system-site-packages: true or false
version: the Python version string

The Python interpreter itself reads it at startup. When Python is launched from inside a venv (via the symlink in bin/), it walks up the directory tree looking for pyvenv.cfg. When found, it sets sys.prefix to the venv directory, redirecting all package lookups to the venv's site-packages.

Level 2 \text{---} Debug and Fix

Scenario: A junior engineer reports that pip install flask succeeded, but import flask raises ModuleNotFoundError. They are confused because which pip shows venv/bin/pip (the venv is activated). Diagnose what likely went wrong and how to verify.

Show Answer

The most likely cause: the engineer ran pip install flask before activating the virtual environment, installing Flask into the system Python. They then activated the venv, which has its own empty site-packages. The venv's pip is now active, but Flask is not installed in the venv.

Verify:

# Check which Python is active
which python
python -c "import sys; print(sys.prefix)"

# Check which pip was used for the install
pip show flask   # will show "not installed" if Flask is not in the venv

# Check if Flask is in system Python
/usr/bin/python3 -c "import flask; print(flask.__file__)"

Fix:

# Make sure the venv is activated
source venv/bin/activate

# Install Flask into the venv
pip install flask

# Verify
python -c "import flask; print(flask.__file__)"
# Should show a path inside venv/

Prevention: Always check which python and which pip after activating a venv before running any install commands.

Level 3 \text{---} Design

Challenge: Your team has three microservices in a monorepo, each requiring different Python versions and different packages:

monorepo/
├── service-a/   # Python 3.10, flask 2.3, sqlalchemy 1.4
├── service-b/   # Python 3.12, fastapi 0.104, sqlalchemy 2.0
└── service-c/   # Python 3.11, django 4.2, celery 5.3

Design the full local development environment setup using pyenv and venv. Include the directory structure, the files to commit to git, the setup commands, and how a new developer joining the team would get their environment running for all three services.

Show Answer

Directory structure:

monorepo/
├── service-a/
│   ├── .python-version          # committed: 3.10.13
│   ├── requirements.txt         # committed: pinned deps
│   ├── requirements-dev.txt     # committed: pinned dev deps
│   ├── venv/                    # gitignored
│   └── src/
├── service-b/
│   ├── .python-version          # committed: 3.12.0
│   ├── requirements.txt
│   ├── requirements-dev.txt
│   ├── venv/
│   └── src/
├── service-c/
│   ├── .python-version          # committed: 3.11.6
│   ├── requirements.txt
│   ├── requirements-dev.txt
│   ├── venv/
│   └── src/
├── .gitignore                   # includes venv/
└── README.md                    # setup instructions

Setup script (scripts/setup.sh):

#!/bin/bash
set -e

# Install required Python versions
pyenv install 3.10.13 --skip-existing
pyenv install 3.11.6 --skip-existing
pyenv install 3.12.0 --skip-existing

setup_service() {
    local service=$1
    echo "Setting up $service..."
    cd "$service"
    # pyenv reads .python-version and sets the correct Python
    python -m venv venv
    source venv/bin/activate
    pip install --upgrade pip
    pip install -r requirements-dev.txt
    deactivate
    cd ..
}

setup_service service-a
setup_service service-b
setup_service service-c

echo "All services set up. Activate individual envs with: source service-X/venv/bin/activate"

New developer onboarding:

git clone <repo>
cd monorepo
brew install pyenv  # or: curl https://pyenv.run | bash
# Add pyenv to shell profile (see pyenv docs)
bash scripts/setup.sh

Key design decisions:

.python-version files are committed so pyenv automatically selects the right Python in each directory
venv/ is gitignored - environments are not committed, they are recreated from pinned requirements.txt
Each service has its own venv, preventing cross-service dependency conflicts
A setup script makes onboarding deterministic and fast

Key Takeaways

A virtual environment is a directory containing a Python symlink/copy and an isolated site-packages - not a separate Python installation
pyvenv.cfg is the file Python reads at startup that redirects sys.prefix to the venv, enabling package isolation
source venv/bin/activate only modifies PATH - isolation is a property of the venv directory structure, not the activation state
pip install targets sys.prefix-relative site-packages; activate the venv before installing
pyenv manages Python versions (3.10, 3.11, 3.12); venv manages packages within a version - they solve different problems and work together
Commit .python-version; never commit venv/
Docker containers provide OS isolation, not Python package isolation; use venv inside multi-stage builds to make artifacts portable between stages
Never sudo pip install - it corrupts the system Python used by OS tooling

What You Will Learn​

Prerequisites​

Part 1 - The Filesystem Structure of a Virtual Environment​

pyvenv.cfg - The Configuration File That Makes a venv a venv​

Part 2 - How Activation Works​

deactivate​

Part 3 \text{---} python -m venv in Production​

Basic Usage​

Useful Flags​

How pip install Finds the Right site-packages​

Part 4 \text{---} virtualenv (Third-Party Alternative)​

Part 5 \text{---} pyenv: Managing Python Versions​

Installation​

Core Commands​

.python-version​

The Complete Project Setup Workflow​

pyenv-virtualenv Plugin​

Part 6 - Docker Containers Are Not Virtual Environments​

Part 7 \text{---} Graded Practice​

Level 1 \text{---} Predict and Identify​

Level 2 \text{---} Debug and Fix​

Level 3 \text{---} Design​

Key Takeaways​

What You Will Learn

Prerequisites

Part 1 - The Filesystem Structure of a Virtual Environment

pyvenv.cfg - The Configuration File That Makes a venv a venv

Part 2 - How Activation Works

deactivate

Part 3 \text{---} python -m venv in Production

Basic Usage

Useful Flags

How pip install Finds the Right site-packages

Part 4 \text{---} virtualenv (Third-Party Alternative)

Part 5 \text{---} pyenv: Managing Python Versions

Installation

Core Commands

.python-version

The Complete Project Setup Workflow

pyenv-virtualenv Plugin

Part 6 - Docker Containers Are Not Virtual Environments

Part 7 \text{---} Graded Practice

Level 1 \text{---} Predict and Identify

Level 2 \text{---} Debug and Fix

Level 3 \text{---} Design

Key Takeaways