Configuration Management - Environment-Driven Apps

This application runs perfectly on the developer's machine. It will explode in production. Can you spot all the configuration problems?

# app.py
import psycopg2
import stripe
import smtplib

# Hardcoded values scattered across the codebase
DB_URL = "postgresql://admin:password123@localhost:5432/myapp"
stripe.api_key = "sk_live_4eC39HqLyjWDarjtT1zdp7dc"
SMTP_HOST = "smtp.gmail.com"
SMTP_PASSWORD = "hunter2"
DEBUG = True
ALLOWED_ORIGINS = ["http://localhost:3000"]
SECRET_KEY = "my-super-secret-key-that-is-not-secret"

def get_db():
    return psycopg2.connect(DB_URL)

Problems: (1) production credentials hardcoded in source code and committed to Git, (2) debug mode enabled with no way to change per environment, (3) no validation - if DB_URL is wrong, the app crashes on first query, not at startup, (4) no separation between dev and production settings. Configuration management solves all of these.

What You Will Learn

• Using environment variables and .env files with python-dotenv
• Building validated, typed configuration with pydantic-settings
• The environment variable hierarchy: defaults, .env, env vars, secrets files
• Managing secrets safely (environment variables vs secret managers)
• Multi-environment configurations (dev, staging, production)
• Configuration as a dependency: injecting settings into services
• The 12-factor config principle and why it matters

Prerequisites

• Experience with Python classes, dataclasses, and type hints
• Familiarity with Pydantic model validation
• Understanding of dependency injection (Lesson 3 in this module)
• Basic experience deploying applications (Docker, cloud, or VPS)

Part 1 - The Configuration Spectrum

Configuration lives on a spectrum from fully hardcoded to fully externalized.

Level	Mechanism	When to Use
Hardcoded	MAX_RETRIES = 3	Truly constant values that never change across environments
Config file	config.py, settings.yaml	Defaults that developers share
.env file	python-dotenv	Local development overrides
Environment variables	os.environ	Production, CI/CD, Docker
Secret manager	AWS Secrets Manager, HashiCorp Vault	Credentials, API keys, certificates

The 12-factor app principle says: store config in the environment. Everything that varies between deploys (dev vs staging vs production) should come from environment variables.

Part 2 - python-dotenv for Local Development

pip install python-dotenv

The .env File

# .env (NEVER commit this file to Git)
DATABASE_URL=postgresql://dev:devpass@localhost:5432/myapp_dev
REDIS_URL=redis://localhost:6379/0
SECRET_KEY=local-dev-secret-not-for-production
STRIPE_API_KEY=sk_test_xxx
SMTP_HOST=localhost
SMTP_PORT=1025
DEBUG=true
LOG_LEVEL=DEBUG
ALLOWED_ORIGINS=http://localhost:3000,http://localhost:5173

Loading Environment Variables

# config.py - basic approach
import os
from dotenv import load_dotenv

# Load .env file into os.environ
load_dotenv()  # looks for .env in current directory
# load_dotenv(".env.local")  # or specify a path

DATABASE_URL = os.getenv("DATABASE_URL", "sqlite:///./dev.db")
REDIS_URL = os.getenv("REDIS_URL", "redis://localhost:6379/0")
SECRET_KEY = os.getenv("SECRET_KEY")
DEBUG = os.getenv("DEBUG", "false").lower() == "true"
LOG_LEVEL = os.getenv("LOG_LEVEL", "INFO")

if SECRET_KEY is None:
    raise ValueError("SECRET_KEY environment variable is required")

The .gitignore Entry

# .gitignore - CRITICAL
.env
.env.local
.env.production
*.pem
*.key

:::danger Never Commit Secrets to Git Even if you later delete the file, Git history retains it forever. If a secret is committed, rotate it immediately. Use git-secrets or pre-commit hooks to prevent accidental commits of .env files. :::

Part 3 - pydantic-settings: Validated, Typed Configuration

python-dotenv gives you strings. pydantic-settings gives you validated, typed, documented configuration with automatic environment variable loading.

pip install pydantic-settings

Basic Usage

# config.py
from pydantic_settings import BaseSettings, SettingsConfigDict
from pydantic import Field, SecretStr, field_validator
from typing import Optional


class Settings(BaseSettings):
    """Application configuration - loaded from environment variables."""

    model_config = SettingsConfigDict(
        env_file=".env",
        env_file_encoding="utf-8",
        case_sensitive=False,  # DATABASE_URL == database_url
        extra="ignore",        # ignore unknown env vars
    )

    # Database
    database_url: str = Field(
        default="sqlite:///./dev.db",
        description="Database connection string",
    )
    database_pool_size: int = Field(default=5, ge=1, le=50)
    database_echo: bool = False

    # Redis
    redis_url: str = "redis://localhost:6379/0"

    # Security
    secret_key: SecretStr  # required - no default
    jwt_algorithm: str = "HS256"
    jwt_expiry_minutes: int = Field(default=30, ge=1, le=1440)

    # External Services
    stripe_api_key: SecretStr = SecretStr("")
    smtp_host: str = "localhost"
    smtp_port: int = Field(default=587, ge=1, le=65535)

    # Application
    debug: bool = False
    log_level: str = "INFO"
    allowed_origins: list[str] = ["http://localhost:3000"]
    app_name: str = "MyApp"
    app_version: str = "0.1.0"

    @field_validator("log_level")
    @classmethod
    def validate_log_level(cls, v: str) -> str:
        valid = {"DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"}
        if v.upper() not in valid:
            raise ValueError(f"log_level must be one of {valid}")
        return v.upper()

    @field_validator("allowed_origins", mode="before")
    @classmethod
    def parse_origins(cls, v):
        if isinstance(v, str):
            return [origin.strip() for origin in v.split(",")]
        return v

How pydantic-settings Resolves Values

Priority order (highest to lowest):

1. Environment variables set in the shell or by Docker/Kubernetes
2. .env file values (only if not already set in environment)
3. Default values specified in the Field() or as class attributes

Using SecretStr for Sensitive Values

settings = Settings()

# SecretStr prevents accidental logging/printing
print(settings.secret_key)
# SecretStr('**********')

# Access the actual value explicitly
actual_key = settings.secret_key.get_secret_value()
# 'my-actual-secret-key'

# In logs, secrets are masked
import logging
logging.info(f"Config loaded: {settings.model_dump()}")
# secret_key='**********', stripe_api_key='**********'

:::tip SecretStr Is a Signal, Not Security SecretStr prevents accidental logging but does not encrypt the value in memory. It is a code hygiene tool - it makes developers explicitly call .get_secret_value(), signaling that they are handling sensitive data intentionally. :::

Part 4 - Startup Validation

The best time to discover a misconfiguration is at application startup, not when the first request arrives.

# main.py
import sys
import logging
from pydantic import ValidationError
from config import Settings

logger = logging.getLogger(__name__)


def create_app():
    # Validate all configuration at startup
    try:
        settings = Settings()
    except ValidationError as e:
        logger.critical(f"Configuration error:\n{e}")
        sys.exit(1)

    logger.info(
        f"Starting {settings.app_name} v{settings.app_version} "
        f"(debug={settings.debug}, log_level={settings.log_level})"
    )

    # Validate derived constraints
    if not settings.debug and settings.secret_key.get_secret_value() == "local-dev-secret":
        logger.critical("Cannot use development secret key in production!")
        sys.exit(1)

    if not settings.debug and settings.database_url.startswith("sqlite"):
        logger.warning("Using SQLite in non-debug mode - not recommended for production")

    return build_fastapi_app(settings)

What Validation Catches

# Missing required field
# SECRET_KEY not set in environment
# pydantic_core._pydantic_core.ValidationError: 1 validation error for Settings
# secret_key
#   Field required [type=missing, input_value={...}]

# Invalid type
# DATABASE_POOL_SIZE=not_a_number
# pydantic_core._pydantic_core.ValidationError: 1 validation error for Settings
# database_pool_size
#   Input should be a valid integer [type=int_parsing, input_value='not_a_number']

# Out of range
# DATABASE_POOL_SIZE=100
# pydantic_core._pydantic_core.ValidationError: 1 validation error for Settings
# database_pool_size
#   Input should be less than or equal to 50 [type=less_than_equal]

# Invalid log level
# LOG_LEVEL=VERBOSE
# pydantic_core._pydantic_core.ValidationError: 1 validation error for Settings
# log_level
#   Value error, log_level must be one of {'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'}

Part 5 - Multi-Environment Configuration

Strategy 1: Environment Variable Prefix

class Settings(BaseSettings):
    model_config = SettingsConfigDict(
        env_prefix="MYAPP_",  # MYAPP_DATABASE_URL, MYAPP_SECRET_KEY, etc.
        env_file=".env",
    )

    database_url: str = "sqlite:///./dev.db"
    secret_key: SecretStr
    debug: bool = False

# .env
MYAPP_DATABASE_URL=postgresql://localhost/myapp
MYAPP_SECRET_KEY=dev-secret
MYAPP_DEBUG=true

This avoids collisions when running multiple applications on the same machine.

Strategy 2: Environment-Specific .env Files

import os

class Settings(BaseSettings):
    model_config = SettingsConfigDict(
        env_file=f".env.{os.getenv('APP_ENV', 'development')}",
        env_file_encoding="utf-8",
    )

    database_url: str
    secret_key: SecretStr
    debug: bool = False

# .env.development
DATABASE_URL=postgresql://dev:devpass@localhost:5432/myapp_dev
SECRET_KEY=dev-secret
DEBUG=true

# .env.staging
DATABASE_URL=postgresql://staging:xxx@staging-db:5432/myapp_staging
SECRET_KEY=staging-secret-rotated-monthly
DEBUG=false

# .env.production (NEVER committed - deployed via CI/CD or secret manager)
DATABASE_URL=postgresql://prod:xxx@prod-db-primary:5432/myapp_prod
SECRET_KEY=production-secret-rotated-weekly
DEBUG=false

# Run in different environments
APP_ENV=development python main.py
APP_ENV=staging python main.py
APP_ENV=production python main.py

Strategy 3: Nested Configuration Classes

class DatabaseSettings(BaseSettings):
    model_config = SettingsConfigDict(env_prefix="DB_")

    url: str = "sqlite:///./dev.db"
    pool_size: int = 5
    pool_timeout: int = 30
    echo: bool = False


class RedisSettings(BaseSettings):
    model_config = SettingsConfigDict(env_prefix="REDIS_")

    url: str = "redis://localhost:6379/0"
    max_connections: int = 10
    socket_timeout: float = 5.0


class EmailSettings(BaseSettings):
    model_config = SettingsConfigDict(env_prefix="EMAIL_")

    smtp_host: str = "localhost"
    smtp_port: int = 587
    from_address: str = "[email protected]"
    username: SecretStr = SecretStr("")
    password: SecretStr = SecretStr("")


class Settings(BaseSettings):
    """Root configuration that composes sub-configurations."""

    model_config = SettingsConfigDict(env_file=".env")

    app_name: str = "MyApp"
    debug: bool = False
    secret_key: SecretStr

    db: DatabaseSettings = DatabaseSettings()
    redis: RedisSettings = RedisSettings()
    email: EmailSettings = EmailSettings()

# .env
SECRET_KEY=my-key
DB_URL=postgresql://localhost/myapp
DB_POOL_SIZE=10
REDIS_URL=redis://cache:6379/1
EMAIL_SMTP_HOST=smtp.sendgrid.net
EMAIL_PASSWORD=SG.xxx

settings = Settings()
print(settings.db.url)        # postgresql://localhost/myapp
print(settings.db.pool_size)  # 10
print(settings.redis.url)     # redis://cache:6379/1

Part 6 - Secrets Management

Level 1: Environment Variables (Minimum Viable Security)

# Set in shell, CI/CD, or Docker
export SECRET_KEY="generated-with-openssl-rand-hex-32"
export DATABASE_URL="postgresql://user:pass@host/db"
export STRIPE_API_KEY="sk_live_xxx"

Pros: simple, works everywhere. Cons: visible in process listings (/proc/*/environ), might be logged.

Level 2: Docker Secrets / Kubernetes Secrets

# Read secrets from mounted files
class Settings(BaseSettings):
    model_config = SettingsConfigDict(
        secrets_dir="/run/secrets",  # Docker secrets mount point
    )

    database_url: str
    secret_key: SecretStr
    stripe_api_key: SecretStr

# Docker Compose
services:
  api:
    image: myapp:latest
    secrets:
      - db_password
      - secret_key

secrets:
  db_password:
    file: ./secrets/db_password.txt
  secret_key:
    file: ./secrets/secret_key.txt

Level 3: External Secret Manager

# adapters/secrets.py
import boto3
import json
from functools import lru_cache


@lru_cache
def get_aws_secrets(secret_name: str, region: str = "us-east-1") -> dict:
    """Fetch secrets from AWS Secrets Manager."""
    client = boto3.client("secretsmanager", region_name=region)
    response = client.get_secret_value(SecretId=secret_name)
    return json.loads(response["SecretString"])


class Settings(BaseSettings):
    model_config = SettingsConfigDict(env_file=".env")

    app_name: str = "MyApp"
    debug: bool = False

    @classmethod
    def from_aws(cls, secret_name: str) -> "Settings":
        """Load settings from AWS Secrets Manager + env vars."""
        secrets = get_aws_secrets(secret_name)
        return cls(
            secret_key=secrets["secret_key"],
            database_url=secrets["database_url"],
            stripe_api_key=secrets.get("stripe_api_key", ""),
        )

:::note Choose the Right Level Solo developer deploying to a VPS? Level 1 (env vars) is fine. Team with CI/CD and Docker? Level 2 (Docker secrets). Enterprise with compliance requirements? Level 3 (Vault or AWS Secrets Manager). Do not over-engineer. :::

Part 7 - Configuration as a Dependency

Settings should be injected, not imported globally. This makes testing easy and configuration explicit.

# dependencies.py
from functools import lru_cache
from fastapi import Depends
from config import Settings


@lru_cache
def get_settings() -> Settings:
    """Singleton settings - loaded once, cached forever."""
    return Settings()


def get_database_url(settings: Settings = Depends(get_settings)) -> str:
    return settings.db.url


def get_stripe_client(settings: Settings = Depends(get_settings)):
    import stripe
    stripe.api_key = settings.stripe_api_key.get_secret_value()
    return stripe

# routes/payments.py
from fastapi import APIRouter, Depends
from config import Settings
from dependencies import get_settings

router = APIRouter()

@router.post("/charge")
def charge(
    amount: int,
    settings: Settings = Depends(get_settings),
):
    # Use settings.stripe_api_key.get_secret_value()
    ...

Testing with Overridden Settings

# tests/conftest.py
import pytest
from fastapi.testclient import TestClient
from config import Settings
from main import app
from dependencies import get_settings


def get_test_settings() -> Settings:
    return Settings(
        database_url="sqlite:///./test.db",
        secret_key="test-secret",
        debug=True,
        stripe_api_key="sk_test_fake",
    )


@pytest.fixture
def client():
    app.dependency_overrides[get_settings] = get_test_settings
    yield TestClient(app)
    app.dependency_overrides.clear()


def test_charge_endpoint(client):
    response = client.post("/charge", json={"amount": 1000})
    assert response.status_code == 200

Part 8 - Configuration Patterns and Anti-Patterns

Pattern: Fail Fast

# GOOD: validate everything at startup
def create_app():
    settings = Settings()  # raises ValidationError if invalid

    # Additional runtime checks
    engine = create_engine(settings.db.url)
    try:
        with engine.connect() as conn:
            conn.execute(text("SELECT 1"))
    except Exception as e:
        raise RuntimeError(f"Cannot connect to database: {e}")

    return build_app(settings)

Anti-Pattern: Late Discovery

# BAD: configuration error discovered 3 hours after deploy
@router.post("/send-email")
def send_email(to: str, body: str):
    host = os.getenv("SMTP_HOST")  # could be None
    port = int(os.getenv("SMTP_PORT"))  # could crash: int(None)
    # This only fails when someone actually tries to send an email

Pattern: Immutable Settings

class Settings(BaseSettings):
    model_config = SettingsConfigDict(frozen=True)  # immutable after creation

    database_url: str
    secret_key: SecretStr

settings = Settings()
settings.database_url = "new_url"  # raises ValidationError - frozen

Anti-Pattern: Global Mutable Config

# BAD: mutable global that anyone can change
config = {"db_url": "postgresql://...", "debug": True}

# Somewhere deep in the code...
config["debug"] = False  # who changed this? when? why?

Pattern: Documented Defaults

class Settings(BaseSettings):
    """
    Application configuration.

    Required environment variables:
        SECRET_KEY: Application secret key for JWT signing
        DATABASE_URL: PostgreSQL connection string

    Optional environment variables:
        DEBUG: Enable debug mode (default: false)
        LOG_LEVEL: Logging level (default: INFO)
        DATABASE_POOL_SIZE: Connection pool size (default: 5, range: 1-50)
    """

    secret_key: SecretStr = Field(description="JWT signing key - generate with `openssl rand -hex 32`")
    database_url: str = Field(description="PostgreSQL connection string")
    debug: bool = Field(default=False, description="Enable debug mode - NEVER true in production")

Part 9 - Configuration for Docker and CI/CD

Docker Compose

# docker-compose.yml
services:
  api:
    build: .
    env_file:
      - .env                  # base defaults
      - .env.${APP_ENV:-development}  # environment overrides
    environment:
      - DATABASE_URL=postgresql://user:pass@db:5432/myapp
      - REDIS_URL=redis://redis:6379/0
    depends_on:
      - db
      - redis

  db:
    image: postgres:16
    environment:
      POSTGRES_DB: myapp
      POSTGRES_USER: user
      POSTGRES_PASSWORD: pass

GitHub Actions

# .github/workflows/test.yml
jobs:
  test:
    runs-on: ubuntu-latest
    env:
      DATABASE_URL: sqlite:///./test.db
      SECRET_KEY: test-secret-for-ci
      DEBUG: "true"
    steps:
      - uses: actions/checkout@v4
      - run: pip install -e ".[test]"
      - run: pytest

Kubernetes ConfigMap and Secret

# configmap.yaml (non-sensitive)
apiVersion: v1
kind: ConfigMap
metadata:
  name: myapp-config
data:
  APP_NAME: "MyApp"
  LOG_LEVEL: "INFO"
  DEBUG: "false"

# secret.yaml (sensitive - base64 encoded)
apiVersion: v1
kind: Secret
metadata:
  name: myapp-secrets
type: Opaque
data:
  SECRET_KEY: bXktc3VwZXItc2VjcmV0LWtleQ==
  DATABASE_URL: cG9zdGdyZXNxbDovL3VzZXI6cGFzc0Bob3N0L2Ri

Key Takeaways

• Externalize everything that varies between environments: database URLs, API keys, feature flags, and log levels should come from environment variables, not hardcoded constants.
• Use pydantic-settings for validated, typed configuration: it loads from environment variables and .env files, validates types and ranges, and masks secrets in logs.
• Fail fast at startup: validate all configuration when the application starts. A ValidationError at startup is worth a hundred cryptic runtime errors.
• Use SecretStr for sensitive values: it prevents accidental logging and signals to developers that the value requires careful handling.
• Inject settings as a dependency: use Depends(get_settings) in FastAPI or constructor injection in services. Never scatter os.getenv() calls across the codebase.
• Match your secrets management to your scale: environment variables for solo projects, Docker secrets for teams, AWS Secrets Manager or Vault for enterprises.
• Never commit .env files: add them to .gitignore and provide a .env.example template with placeholder values.

Graded Practice Challenges

Level 1 - Identify the Problem

Question 1: What is wrong with this configuration loading?

import os

DATABASE_URL = os.environ["DATABASE_URL"]
POOL_SIZE = os.environ.get("POOL_SIZE", 5)

Answer

Two issues: (1) os.environ["DATABASE_URL"] raises KeyError at import time if the variable is not set, but with no helpful error message explaining what is needed. (2) POOL_SIZE will be the string "5", not the integer 5. Using os.environ.get() always returns strings, and no type conversion or validation is performed. With pydantic-settings, both issues are solved - required fields give clear error messages and types are automatically coerced and validated.

Question 2: Why is frozen=True important for settings objects?

Answer

frozen=True makes the settings object immutable after creation. This prevents any code from accidentally (or intentionally) modifying configuration at runtime, such as settings.debug = True to bypass a check. Immutable settings are easier to reason about because the configuration loaded at startup is guaranteed to be the same configuration used throughout the application's lifetime.

Question 3: A developer commits a .env file, realizes the mistake, deletes it, and pushes again. Is the problem solved?

Answer

No. Git retains the file in its history. Anyone with access to the repository can check out the old commit and read the secrets. The developer must: (1) immediately rotate all credentials that were in the .env file, (2) use git filter-branch or BFG Repo-Cleaner to remove the file from history, (3) force-push the cleaned history, and (4) add .env to .gitignore and set up pre-commit hooks to prevent future accidents.

Level 2 - Refactoring Challenge

Refactor this scattered configuration into a proper pydantic-settings setup:

# views.py
import os
db = connect(os.getenv("DB_URL", "sqlite:///dev.db"))

# email.py
SMTP = os.environ.get("SMTP_HOST", "localhost")
PORT = int(os.environ.get("SMTP_PORT", "587"))

# auth.py
SECRET = os.getenv("JWT_SECRET")  # crashes if not set at runtime
ALGO = "HS256"
EXPIRY = int(os.getenv("JWT_EXPIRY", "30"))

# stripe_client.py
import stripe
stripe.api_key = os.getenv("STRIPE_KEY", "sk_test_xxx")

Produce: (a) a Settings class with nested sub-configs, (b) validation rules (port range, expiry range, required fields), (c) a .env.example file, (d) a FastAPI dependency that provides settings.

Level 3 - Design Challenge

Design the configuration management strategy for a multi-tenant SaaS application where:

• Each tenant has different database credentials
• Some tenants have custom SMTP servers
• Feature flags vary by tenant
• API rate limits are tenant-specific
• Secrets are stored in AWS Secrets Manager

How do you structure the configuration? Is it one Settings object or a hierarchy? How do per-tenant overrides work? How do you cache and refresh secrets that rotate?

What's Next

In the next lesson, The 12-Factor App - Building Deployable Python Apps, we will explore all 12 factors of the 12-factor methodology with Python-specific examples, showing how the configuration principles from this lesson fit into the complete picture of building production-deployable applications.