Secrets Management - Never Hardcode Credentials
Before you read any further, search this Python file for secrets. How many can you find?
import psycopg2
import requests
API_KEY = "openai-api-key-goes-here"
STRIPE_SECRET = "stripe-live-key-goes-here"
conn = psycopg2.connect(DATABASE_URL)
response = requests.get(
"https://api.openai.com/v1/models",
headers={"Authorization": f"Bearer {API_KEY}"},
)
There are three hardcoded secrets in this file. If this code is committed to version control, every person who has ever had access to the repository - plus anyone who gains access in the future - has production database credentials, an OpenAI API key, and a live Stripe secret key. This lesson teaches you how to prevent this scenario entirely.
What You Will Learn
- Why hardcoded credentials are the most common source of breaches
- How to use python-dotenv and
.envfiles for local development - How to load secrets from environment variables with
os.environ - How Pydantic SecretStr prevents accidental secret logging
- How to use AWS Secrets Manager and HashiCorp Vault in Python
- How to detect leaked secrets with git-secrets and gitleaks
- Essential
.gitignorepatterns for secret files - How to implement secret rotation without downtime
Prerequisites
- Python environment management, virtual environments
- FastAPI configuration patterns (from Intermediate course)
- Basic understanding of environment variables
pip install python-dotenv pydantic-settings boto3 hvac
Part 1 - The Cost of a Hardcoded Secret
When a secret is committed to Git, removing it is nearly impossible:
Real-world incident timelines show that exposed AWS keys on GitHub are exploited within minutes by automated scanners. The attacker spins up cryptocurrency mining instances, and the victim receives a bill for thousands of dollars.
If a secret is ever committed to Git, consider it compromised. Even if you remove it in the next commit, it remains in the Git history. The secret must be rotated immediately.
Part 2 - python-dotenv: Local Development Secrets
The .env file pattern separates configuration from code. Secrets are stored in a .env file that is never committed to version control:
# .env (in project root - NEVER committed to Git)
DATABASE_URL=postgresql://admin:localdev123@localhost:5432/engineersofai
JWT_SECRET_KEY=dev-only-secret-not-for-production-use
OPENAI_API_KEY=sk-proj-your-key-here...
KEYCLOAK_CLIENT_SECRET=dev-keycloak-secret
PASSWORD_PEPPER=0a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d
STRIPE_SECRET_KEY=sk_test_51ABC...
# config.py
import os
from dotenv import load_dotenv
# Load .env file into environment variables
load_dotenv() # Reads .env from current directory or parents
# Access secrets via os.environ
DATABASE_URL = os.environ["DATABASE_URL"] # Raises KeyError if missing
JWT_SECRET = os.environ.get("JWT_SECRET_KEY", "") # Returns "" if missing
# Validate that required secrets are present
REQUIRED_SECRETS = [
"DATABASE_URL",
"JWT_SECRET_KEY",
"OPENAI_API_KEY",
]
def validate_secrets():
"""Fail fast at startup if required secrets are missing."""
missing = [s for s in REQUIRED_SECRETS if not os.environ.get(s)]
if missing:
raise RuntimeError(
f"Missing required environment variables: {', '.join(missing)}"
)
validate_secrets()
The .env.example Pattern
Provide a template file showing which variables are needed, without actual values:
# .env.example (committed to Git - documents required config)
DATABASE_URL=postgresql://user:password@localhost:5432/dbname
JWT_SECRET_KEY=generate-a-random-32-byte-hex-string
OPENAI_API_KEY=sk-proj-your-key-here
KEYCLOAK_CLIENT_SECRET=your-keycloak-client-secret
PASSWORD_PEPPER=generate-a-random-32-byte-hex-string
STRIPE_SECRET_KEY=sk_test_your-key-here
# Generate secure random secrets for development
import os
print(f"JWT_SECRET_KEY={os.urandom(32).hex()}")
print(f"PASSWORD_PEPPER={os.urandom(32).hex()}")
# JWT_SECRET_KEY=a3b1c4d5e6f7...
# PASSWORD_PEPPER=1f2e3d4c5b6a...
Part 3 - Pydantic Settings and SecretStr
Pydantic's SecretStr type prevents secrets from appearing in logs, repr output, JSON serialization, and error messages:
from pydantic import SecretStr
from pydantic_settings import BaseSettings
class Settings(BaseSettings):
# Regular config - visible in logs
app_name: str = "EngineersOfAI"
debug: bool = False
database_host: str = "localhost"
database_port: int = 5432
database_name: str = "engineersofai"
# Secrets - masked in output
database_password: SecretStr
jwt_secret_key: SecretStr
openai_api_key: SecretStr
keycloak_client_secret: SecretStr
password_pepper: SecretStr
class Config:
env_file = ".env"
env_file_encoding = "utf-8"
settings = Settings()
# SecretStr masks the value in repr and str
print(settings.database_password)
# SecretStr('**********')
print(repr(settings))
# Settings(app_name='EngineersOfAI', ..., database_password=SecretStr('**********'), ...)
# Access the actual value when needed
actual_password = settings.database_password.get_secret_value()
print(actual_password)
# SuperSecret123!
# JSON serialization masks secrets
print(settings.model_dump_json())
# {"app_name": "EngineersOfAI", ..., "database_password": "**********", ...}
Building the Database URL Safely
from pydantic import SecretStr, computed_field
from pydantic_settings import BaseSettings
class Settings(BaseSettings):
database_host: str = "localhost"
database_port: int = 5432
database_name: str = "engineersofai"
database_user: str = "app"
database_password: SecretStr
@computed_field
@property
def database_url(self) -> str:
"""Build the database URL from components."""
password = self.database_password.get_secret_value()
return (
f"postgresql+asyncpg://{self.database_user}:{password}"
f"@{self.database_host}:{self.database_port}"
f"/{self.database_name}"
)
class Config:
env_file = ".env"
Never log the Settings object or include it in error reports without redaction. Even with SecretStr, a developer who calls .get_secret_value() and then logs the result will expose the secret. Establish a team convention that .get_secret_value() is only called at the point of use (database connection, API call), never stored in a plain string variable.
Part 4 - AWS Secrets Manager
For production environments, dedicated secrets managers provide encryption, access control, rotation, and audit logging:
import json
import boto3
from functools import lru_cache
def get_secret(secret_name: str, region: str = "us-east-1") -> dict:
"""Retrieve a secret from AWS Secrets Manager."""
client = boto3.client("secretsmanager", region_name=region)
response = client.get_secret_value(SecretId=secret_name)
secret_string = response["SecretString"]
return json.loads(secret_string)
# Usage
db_secret = get_secret("prod/engineersofai/database")
# Returns: {"username": "app", "password": "...", "host": "...", "port": 5432}
DATABASE_URL = (
f"postgresql+asyncpg://{db_secret['username']}:{db_secret['password']}"
f"@{db_secret['host']}:{db_secret['port']}/engineersofai"
)
Caching Secrets with TTL
Secrets should be cached to avoid API calls on every request, but refreshed periodically for rotation:
import time
import json
import boto3
class SecretsCache:
"""Cache secrets with automatic refresh."""
def __init__(self, region: str = "us-east-1", ttl: int = 300):
self.client = boto3.client("secretsmanager", region_name=region)
self.ttl = ttl # Refresh every 5 minutes
self._cache: dict[str, tuple[dict, float]] = {}
def get(self, secret_name: str) -> dict:
cached = self._cache.get(secret_name)
if cached:
value, timestamp = cached
if time.time() - timestamp < self.ttl:
return value
# Fetch from AWS
response = self.client.get_secret_value(SecretId=secret_name)
value = json.loads(response["SecretString"])
self._cache[secret_name] = (value, time.time())
return value
secrets = SecretsCache(ttl=300)
# In your application:
def get_database_url() -> str:
db = secrets.get("prod/engineersofai/database")
return f"postgresql+asyncpg://{db['username']}:{db['password']}@{db['host']}/engineersofai"
Part 5 - HashiCorp Vault
HashiCorp Vault is an open-source secrets manager with dynamic secret generation, lease management, and fine-grained access control:
import hvac
# Initialize Vault client
client = hvac.Client(
url="https://vault.engineersofai.com:8200",
token=os.environ["VAULT_TOKEN"], # Token itself from env var
)
# Verify connection
assert client.is_authenticated()
# Read a static secret (KV v2 engine)
secret = client.secrets.kv.v2.read_secret_version(
mount_point="secret",
path="engineersofai/database",
)
db_creds = secret["data"]["data"]
# {'username': 'app', 'password': '...', 'host': 'db.internal'}
# Read a dynamic secret (database engine)
# Vault generates a temporary database credential with a lease
dynamic_creds = client.secrets.database.generate_credentials(
name="engineersofai-readonly",
mount_point="database",
)
temp_username = dynamic_creds["data"]["username"]
temp_password = dynamic_creds["data"]["password"]
lease_duration = dynamic_creds["lease_duration"] # e.g., 3600 seconds
# These credentials automatically expire after the lease duration
Vault AppRole Authentication (Production)
In production, do not use a static token. Use AppRole for machine authentication:
import hvac
import os
client = hvac.Client(url="https://vault.engineersofai.com:8200")
# Authenticate with AppRole
role_id = os.environ["VAULT_ROLE_ID"]
secret_id = os.environ["VAULT_SECRET_ID"]
auth_response = client.auth.approle.login(
role_id=role_id,
secret_id=secret_id,
)
client.token = auth_response["auth"]["client_token"]
# Now read secrets
secret = client.secrets.kv.v2.read_secret_version(
mount_point="secret",
path="engineersofai/api-keys",
)
Part 6 - Preventing Secret Commits with git-secrets and gitleaks
git-secrets (AWS-maintained)
# Install git-secrets
brew install git-secrets # macOS
# or
pip install git-secrets
# Set up in your repository
cd /path/to/repo
git secrets --install
git secrets --register-aws # Add AWS key patterns
# Add custom patterns
git secrets --add 'sk-proj-[a-zA-Z0-9]{20,}' # OpenAI keys
git secrets --add 'sk_live_[a-zA-Z0-9]{20,}' # Stripe live keys
git secrets --add 'password\s*=\s*["\x27][^"\x27]{8,}' # Hardcoded passwords
# Now git commit will be blocked if secrets are detected:
# git commit -m "add config"
# ERROR: Matched one or more prohibited patterns
# Possible mitigations:
# - Mark false positives: git secrets --add --allowed '...'
gitleaks (Comprehensive Scanner)
# Install gitleaks
brew install gitleaks
# Scan current state of the repo
gitleaks detect --source . --verbose
# Scan Git history for previously committed secrets
gitleaks detect --source . --verbose --log-opts="--all"
# Scan staged changes (pre-commit hook)
gitleaks protect --staged --verbose
Pre-Commit Hook Integration
# .pre-commit-config.yaml
repos:
- repo: https://github.com/gitleaks/gitleaks
rev: v8.18.0
hooks:
- id: gitleaks
- repo: https://github.com/awslabs/git-secrets
rev: master
hooks:
- id: git-secrets
# Install pre-commit hooks
pip install pre-commit
pre-commit install
# Now every commit is scanned for secrets
# git commit -m "update config"
# gitleaks...........................................................Failed
# hookid: gitleaks
# Finding: sk-proj-your-key-heredef456ghi789jkl012mno345pqr678
# File: config.py
# Line: 5
git-secrets and gitleaks are the last line of defense, not the first. They cannot catch every possible secret format. The primary defense is never putting secrets in code files - always use environment variables or a secrets manager. These tools catch mistakes that slip through.
Part 7 - .gitignore Patterns for Secrets
# .gitignore - essential patterns for secret files
# Environment files
.env
.env.local
.env.production
.env.staging
.env.*.local
# Do NOT ignore .env.example - it documents required variables
# !.env.example
# Python
*.pyc
__pycache__/
# Key files
*.pem
*.key
*.p12
*.pfx
*.jks
*.keystore
# AWS
.aws/credentials
.aws/config
# Docker secrets
docker-compose.override.yml
# IDE secrets
.idea/
.vscode/settings.json
# Terraform state (contains secrets)
*.tfstate
*.tfstate.backup
.terraform/
# Vault tokens
.vault-token
# Certificate files
*.crt
*.cert
*.ca-bundle
Verifying .gitignore Works
# verify_gitignore.py
import subprocess
from pathlib import Path
SENSITIVE_PATTERNS = [
".env",
"*.pem",
"*.key",
"credentials*",
"*secret*",
]
def check_tracked_secrets():
"""Check if any sensitive files are tracked by Git."""
result = subprocess.run(
["git", "ls-files"],
capture_output=True,
text=True,
)
tracked_files = result.stdout.strip().splitlines()
warnings = []
for filepath in tracked_files:
name = Path(filepath).name.lower()
for pattern in SENSITIVE_PATTERNS:
# Simple pattern matching
if pattern.startswith("*"):
if name.endswith(pattern[1:]):
warnings.append(f"WARNING: Tracked file matches '{pattern}': {filepath}")
elif name == pattern:
warnings.append(f"WARNING: Tracked file matches '{pattern}': {filepath}")
return warnings
if __name__ == "__main__":
warnings = check_tracked_secrets()
for w in warnings:
print(w)
if not warnings:
print("No sensitive files found in Git tracking")
Part 8 - Secret Rotation
Secrets should be rotated regularly and immediately upon suspected compromise. The challenge is rotating without downtime:
Strategy 1: Dual-Read During Rotation
import os
from passlib.context import CryptContext
class RotatableSecret:
"""Support two active secrets during rotation."""
def __init__(self, env_var: str):
self.current = os.environ[env_var]
self.previous = os.environ.get(f"{env_var}_PREVIOUS", "")
def verify_hmac(self, data: str, signature: str) -> bool:
"""Try current key first, then previous."""
import hmac
import hashlib
# Try current key
expected = hmac.new(
self.current.encode(), data.encode(), hashlib.sha256
).hexdigest()
if hmac.compare_digest(expected, signature):
return True
# Try previous key (during rotation window)
if self.previous:
expected_prev = hmac.new(
self.previous.encode(), data.encode(), hashlib.sha256
).hexdigest()
if hmac.compare_digest(expected_prev, signature):
return True
return False
# Environment:
# JWT_SECRET_KEY=new-secret-key-2024
# JWT_SECRET_KEY_PREVIOUS=old-secret-key-2023
jwt_secret = RotatableSecret("JWT_SECRET_KEY")
Strategy 2: AWS Secrets Manager Automatic Rotation
import boto3
def setup_rotation(secret_name: str, lambda_arn: str):
"""Configure automatic rotation for a secret."""
client = boto3.client("secretsmanager")
client.rotate_secret(
SecretId=secret_name,
RotationLambdaARN=lambda_arn,
RotationRules={
"AutomaticallyAfterDays": 30, # Rotate every 30 days
},
)
Rotation Process
Part 9 - Real-World: Managing Secrets in a FastAPI Application
A complete production configuration pattern:
# app/config.py
import os
from functools import lru_cache
from pydantic import SecretStr, computed_field
from pydantic_settings import BaseSettings, SettingsConfigDict
class Settings(BaseSettings):
model_config = SettingsConfigDict(
env_file=".env",
env_file_encoding="utf-8",
case_sensitive=False,
)
# --- Application Config (non-sensitive) ---
app_name: str = "EngineersOfAI"
app_env: str = "development" # development, staging, production
debug: bool = False
host: str = "0.0.0.0"
port: int = 8001
# --- Database (mixed) ---
database_host: str = "localhost"
database_port: int = 5432
database_name: str = "engineersofai"
database_user: str = "app"
database_password: SecretStr = SecretStr("")
# --- Auth (sensitive) ---
jwt_secret_key: SecretStr = SecretStr("")
jwt_algorithm: str = "RS256"
keycloak_realm_url: str = "http://localhost:8080/realms/main"
keycloak_client_id: str = "engineersofai-api"
keycloak_client_secret: SecretStr = SecretStr("")
# --- External APIs (sensitive) ---
openai_api_key: SecretStr = SecretStr("")
stripe_secret_key: SecretStr = SecretStr("")
# --- Security ---
password_pepper: SecretStr = SecretStr("")
session_secret: SecretStr = SecretStr("")
@computed_field
@property
def database_url(self) -> str:
password = self.database_password.get_secret_value()
return (
f"postgresql+asyncpg://{self.database_user}:{password}"
f"@{self.database_host}:{self.database_port}"
f"/{self.database_name}"
)
@computed_field
@property
def is_production(self) -> bool:
return self.app_env == "production"
@lru_cache
def get_settings() -> Settings:
"""Cached settings instance - loaded once at startup."""
return Settings()
# --- In FastAPI ---
from fastapi import FastAPI, Depends
app = FastAPI()
@app.on_event("startup")
async def startup():
settings = get_settings()
if settings.is_production:
# Validate all production secrets are present
required = [
settings.database_password,
settings.jwt_secret_key,
settings.keycloak_client_secret,
settings.password_pepper,
settings.session_secret,
]
for secret in required:
if not secret.get_secret_value():
raise RuntimeError("Missing required secret in production")
# Log config without secrets
print(f"Starting {settings.app_name} in {settings.app_env} mode")
print(f"Database: {settings.database_host}:{settings.database_port}/{settings.database_name}")
# Secrets are NOT logged - SecretStr masks them
print(f"JWT Secret: {settings.jwt_secret_key}") # Prints: SecretStr('**********')
Docker Compose Secrets
# docker-compose.yml
version: "3.8"
services:
api:
build: .
env_file:
- .env # Local development only
environment:
- APP_ENV=development
secrets:
- db_password
- jwt_secret
api-prod:
build: .
environment:
- APP_ENV=production
- DATABASE_HOST=db.prod.internal
secrets:
- db_password
- jwt_secret
secrets:
db_password:
file: ./secrets/db_password.txt # NOT committed to Git
jwt_secret:
file: ./secrets/jwt_secret.txt
# Reading Docker secrets in Python
def read_docker_secret(secret_name: str) -> str:
"""Read a secret from Docker's /run/secrets/ mount."""
secret_path = f"/run/secrets/{secret_name}"
try:
with open(secret_path, "r") as f:
return f.read().strip()
except FileNotFoundError:
return ""
Key Takeaways
- Never hardcode secrets in source code, configuration files, or Docker images
- If a secret is committed to Git, consider it compromised and rotate immediately
- Use python-dotenv and
.envfiles for local development, with.env.exampleas documentation - Use Pydantic SecretStr to prevent secrets from appearing in logs, repr, and JSON
- Use
os.environwith fail-fast validation at startup - do not let missing secrets cause runtime errors - In production, use a secrets manager (AWS Secrets Manager, Vault, or GCP Secret Manager)
- Install git-secrets or gitleaks as pre-commit hooks to catch accidental commits
- Maintain a comprehensive
.gitignorethat covers all secret file patterns - Implement secret rotation with dual-read support for zero-downtime transitions
- Separate configuration (host, port, app name) from secrets (passwords, keys, tokens)
Graded Practice Challenges
Level 1 - Identify the Vulnerability
Question 1: What is wrong with this Dockerfile?
FROM python:3.12-slim
ENV DATABASE_URL=postgresql://admin:prod_password@db:5432/myapp
ENV API_KEY=sk-proj-your-key-here
COPY . .
CMD ["uvicorn", "app:app"]
Answer
Secrets are baked into the Docker image via ENV instructions. Anyone who pulls the image can inspect it with docker inspect or docker history and see all environment variables, including the database password and API key. Even if you later override the ENV at runtime, the secret remains in the image layer. Use runtime-injected environment variables (docker run -e) or Docker secrets instead.
Question 2: A developer uses print(f"Connecting to {settings.database_url}") at startup. What is the risk?
Answer
The database_url computed field includes the password in the connection string. This prints the full URL with credentials to stdout/stderr, which typically goes to log files, log aggregation services (CloudWatch, Datadog), and potentially error reporting tools. Anyone with log access sees the database password. Log the host and database name separately, never the full connection URL.
Question 3: What is the problem with this rotation strategy?
# Step 1: Update the secret in Vault
# Step 2: Restart all application instances
# Step 3: Revoke the old secret
Answer
There is a gap between step 1 and step 2 where running instances still use the old secret. If step 3 (revoke old secret) happens before step 2 (restart) completes, instances that have not restarted will fail. The correct order is: (1) Create the new secret alongside the old one. (2) Deploy application code that accepts both old and new secrets. (3) Restart instances (they now use the new secret). (4) After all instances are running with the new secret, revoke the old one.
Level 2 - Fix the Vulnerability
This configuration module has multiple secrets management issues. Fix all of them:
# config.py
import os
DATABASE_URL = "postgresql://admin:password123@localhost:5432/mydb"
API_KEY = "sk-proj-my-openai-key-here"
JWT_SECRET = "jwt-secret"
STRIPE_KEY = os.environ.get("STRIPE_KEY", "sk_live_default_key")
def get_config():
return {
"database_url": DATABASE_URL,
"api_key": API_KEY,
"jwt_secret": JWT_SECRET,
"stripe_key": STRIPE_KEY,
}
# In app startup
config = get_config()
print(f"Config loaded: {config}")
Solution
# config.py
import os
import logging
from pydantic import SecretStr
from pydantic_settings import BaseSettings
logger = logging.getLogger(__name__)
class Settings(BaseSettings):
database_host: str = "localhost"
database_port: int = 5432
database_name: str = "mydb"
database_user: str = "app"
database_password: SecretStr
openai_api_key: SecretStr
jwt_secret_key: SecretStr
stripe_secret_key: SecretStr
class Config:
env_file = ".env"
@property
def database_url(self) -> str:
pw = self.database_password.get_secret_value()
return (
f"postgresql+asyncpg://{self.database_user}:{pw}"
f"@{self.database_host}:{self.database_port}"
f"/{self.database_name}"
)
settings = Settings()
# Safe logging - secrets are masked
logger.info(
"Config loaded: host=%s, port=%d, db=%s",
settings.database_host,
settings.database_port,
settings.database_name,
)
# Secrets are NOT logged
Fixes: (1) All secrets via environment variables with SecretStr. (2) No default values for secrets. (3) No hardcoded credentials. (4) Logging does not include secrets. (5) Settings validated at import time - missing secrets cause immediate failure.
Level 3 - Design a Secure System
Design a secrets management architecture for a microservices platform with:
- 15 microservices, each needing database credentials and inter-service API keys
- Deployment to Kubernetes on AWS EKS
- Compliance requirement: secrets must be rotated every 90 days
- Audit requirement: every secret access must be logged
- Development, staging, and production environments with different secrets
Document your approach for: secret storage, secret injection into containers, rotation without downtime, access control (which service can access which secrets), audit logging, and emergency secret revocation.
Design Hints
- Storage: AWS Secrets Manager for all secrets. Organized by path:
prod/service-name/database,staging/service-name/api-keys. - Injection: Use the AWS Secrets Manager CSI driver for Kubernetes. Secrets are mounted as files in the pod filesystem, not environment variables (env vars are visible in
kubectl describe pod). - Rotation: AWS Secrets Manager automatic rotation with Lambda functions. Use dual-read pattern in application code. Lambda creates new credential, tests it, promotes it, and cleans up the old one.
- Access control: IAM roles per service (Kubernetes IRSA). Each service's IAM role only has
secretsmanager:GetSecretValuepermission for its own secrets path. - Audit: AWS CloudTrail logs every
GetSecretValueAPI call with the caller's identity, timestamp, and secret ARN. Alert on unexpected access patterns. - Emergency revocation: One-click script that rotates all secrets for a compromised service, restarts its pods, and alerts the security team.
- Dev/staging: Use separate AWS accounts for environment isolation. Developers use local
.envfiles with test credentials, never production secrets.
What's Next
In the next lesson, Secure Coding Patterns, you will bring everything together - applying defense-in-depth principles, CORS configuration, rate limiting, dependency auditing, and static security analysis to harden an entire FastAPI application.
