Publishing Packages - From Source to PyPI

Reading time: ~28 minutes | Level: Intermediate → Engineering

Before reading further, consider what actually happens in those 2 seconds when you run:

pip install requests
# 1. pip contacts https://pypi.org/simple/requests/ (the Simple API)
# 2. Gets a list of all available distributions for 'requests'
# 3. Filters by: Python version, ABI, platform, version constraints
# 4. Selects: requests-2.31.0-py3-none-any.whl (prefers wheel over sdist)
# 5. Downloads the .whl file (~62 KB)
# 6. Verifies SHA-256 hash against PyPI's recorded hash
# 7. Extracts the wheel into site-packages/ (no build step)
# 8. Done

# But what IS a wheel? How did it get on PyPI?
# And if you wrote requests - how would you put it there?

The wheel file on PyPI did not appear by magic. Someone wrote source code, configured a build system, ran a build tool, authenticated with PyPI's API, uploaded the files, and verified the result. This lesson is that process, end to end, at engineering depth.

What You Will Learn

• Distribution formats: sdist and wheel - what they are, when each is used
• Wheel filename anatomy: how to read what a wheel supports
• Building with python -m build
• The TestPyPI staging environment and why it exists
• Uploading with twine and poetry publish
• Authentication: API tokens and how to use them securely
• PyPI project page: classifiers, keywords, and README
• Private package registries: GitLab, AWS CodeArtifact, Nexus
• Consuming from private registries
• Automating releases with CI/CD: GitLab CI and GitHub Actions OIDC

Prerequisites

• Lesson 03 (pyproject.toml) - building requires a valid pyproject.toml
• Lesson 04 (Poetry) - covers poetry publish as an alternative to twine
• Lesson 05 (Semantic Versioning) - release tagging drives automated publish pipelines

Part 1 - Distribution Formats: sdist vs Wheel

sdist - Source Distribution

An sdist (.tar.gz) is an archive of your source code plus build instructions. When pip installs an sdist, it must:

1. Extract the archive
2. Run the build backend (hatchling, setuptools, flit-core, etc.)
3. Compile any C/Cython/Rust extensions
4. Produce a wheel internally
5. Install the wheel

sdist installs are slower and require build tools (a C compiler, Rust toolchain, etc.) for packages with compiled extensions. For pure-Python packages, the "build" step is trivial - but the overhead still exists.

# sdist: what it looks like on disk
requests-2.31.0.tar.gz
├── PKG-INFO              ← package metadata
├── setup.cfg             ← (legacy) configuration
├── pyproject.toml
├── src/
│   └── requests/
│       ├── __init__.py
│       ├── adapters.py
│       ├── auth.py
│       └── ...
└── tests/
    └── ...

Wheel - Pre-Built Distribution

A wheel (.whl) is a zip file containing the package in its final installed form. pip installs a wheel by simply extracting it into site-packages/. No build step, no compiler, no build tools needed.

requests-2.31.0-py3-none-any.whl
├── requests/
│   ├── __init__.py
│   ├── adapters.py
│   └── ...
├── requests-2.31.0.dist-info/
│   ├── METADATA          ← package metadata
│   ├── WHEEL             ← wheel format info
│   ├── RECORD            ← file hashes for integrity verification
│   └── INSTALLER

pip always prefers wheels when available. sdists are the fallback when no compatible wheel exists.

Part 2 - Wheel Filename Anatomy

The wheel filename encodes its compatibility:

{distribution}-{version}-{python_tag}-{abi_tag}-{platform_tag}.whl

Pure Python Wheel (Universal)

requests-2.31.0-py3-none-any.whl
│         │      │    │      │
│         │      │    │      └── platform: any (works on Linux, macOS, Windows)
│         │      │    └───────── ABI: none (no C extensions, no ABI dependency)
│         │      └────────────── Python: py3 (pure Python 3, any implementation)
│         └───────────────────── version: 2.31.0
└─────────────────────────────── distribution: requests

py3-none-any means: pure Python 3, no compiled extensions, works everywhere. This is the wheel format for pure-Python packages.

CPython-Specific Wheel (With C Extension)

numpy-1.26.0-cp312-cp312-manylinux_2_17_x86_64.whl
│      │       │      │     │
│      │       │      │     └── platform: manylinux 2.17, x86_64 Linux
│      │       │      └──────── ABI: CPython 3.12 ABI
│      │       └─────────────── Python: CPython 3.12
│      └─────────────────────── version: 1.26.0
└────────────────────────────── distribution: numpy

cp312-cp312-manylinux_2_17_x86_64 means: CPython 3.12 specifically, compiled for Linux x86_64 with glibc 2.17+. This wheel only installs on CPython 3.12 on compatible Linux distributions.

Common Platform Tags

Platform tag	What it covers
any	All platforms (pure Python only)
win_amd64	Windows 64-bit
macosx_14_0_arm64	macOS 14+ Apple Silicon
macosx_12_0_x86_64	macOS 12+ Intel
manylinux_2_17_x86_64	Most Linux x86_64 distributions (glibc >=2.17)
manylinux_2_17_aarch64	Most Linux ARM64 distributions

note

Pure Python wheels (py3-none-any) work everywhere and are built once. Compiled extensions (C, Cython, Rust) need separate wheels for each platform and Python version combination. Building multi-platform wheels is done with cibuildwheel, which runs on CI across Linux, macOS, and Windows runners to produce the full matrix of compatible wheels. For pure Python packages, you never need to worry about this.

Part 3 - Building with python -m build

The build package is the PEP 517-compliant frontend for building distributions. It works with any pyproject.toml-configured build backend.

# Install the build tool
pip install build

# Build both sdist and wheel (recommended - always publish both)
python -m build

# Output:
# dist/
#   my-package-1.0.0.tar.gz       ← sdist
#   my-package-1.0.0-py3-none-any.whl  ← wheel

The pyproject.toml specifies which build backend to use:

# Using hatchling (recommended for new projects)
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

# Using Poetry
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

# Using Flit (simple, for pure Python)
[build-system]
requires = ["flit_core>=3.2"]
build-backend = "flit_core.build"

# Using setuptools (legacy, widely compatible)
[build-system]
requires = ["setuptools>=68", "wheel"]
build-backend = "setuptools.build_meta"

# Build only a wheel (faster, no sdist)
python -m build --wheel

# Build only sdist
python -m build --sdist

# Build in a temporary isolated environment (default behavior)
# This ensures your build does not accidentally depend on your current environment
python -m build

# Build without isolation (faster, but risks environment contamination)
python -m build --no-isolation

Part 4 - TestPyPI: The Staging Environment

TestPyPI (test.pypi.org) is a separate instance of PyPI for testing your publish workflow before releasing to the real index. It has its own accounts, package namespace, and data.

Why TestPyPI exists:

• PyPI package releases are permanent: you cannot delete or overwrite a release
• PyPI package names are globally unique and permanent: if you publish myutils, that name is reserved forever
• Testing your pyproject.toml metadata, classifiers, and README rendering should not consume a real release slot

tip

Always upload to TestPyPI first. Verify that: (1) the metadata renders correctly on the project page, (2) the README displays properly as the long description, (3) pip install --index-url https://test.pypi.org/simple/ yourpackage works as expected. Only then upload to the real PyPI. You cannot undo a real PyPI release.

# Register at https://test.pypi.org/ (separate account from pypi.org)
# Create an API token at: https://test.pypi.org/manage/account/token/

# Upload to TestPyPI
twine upload --repository testpypi dist/*

# Verify the install from TestPyPI
pip install \
  --index-url https://test.pypi.org/simple/ \
  --extra-index-url https://pypi.org/simple/ \
  yourpackage

# --extra-index-url pypi.org/simple/ is needed because TestPyPI
# does not have all the dependencies your package requires

Part 5 - Uploading with Twine

twine is the standard tool for uploading distributions to PyPI.

pip install twine

# Always check before uploading
twine check dist/*
# Checks: metadata validity, README rendering, required fields
# Output example:
# Checking dist/my-package-1.0.0.tar.gz: PASSED
# Checking dist/my-package-1.0.0-py3-none-any.whl: PASSED

# Upload to PyPI
twine upload dist/*

# Upload to TestPyPI
twine upload --repository testpypi dist/*

# Upload with explicit API token
twine upload --username __token__ --password pypi-AgE... dist/*

Authentication: API Tokens

Never use your PyPI username and password with twine. Use API tokens instead:

1. Log in to pypi.org
2. Go to Account Settings → API tokens
3. Create a token scoped to a specific project (not "Entire account")
4. Copy the token - it starts with pypi-

# Option A: Environment variables (best for CI)
export TWINE_USERNAME=__token__
export TWINE_PASSWORD=pypi-AgEIcGkg...
twine upload dist/*

# Option B: ~/.pypirc file (for local use)
# The username is always __token__ when using tokens

# ~/.pypirc
[distutils]
index-servers =
    pypi
    testpypi

[pypi]
username = __token__
password = pypi-AgEIcGkg...

[testpypi]
repository = https://test.pypi.org/legacy/
username = __token__
password = pypi-xxxx...

danger

Never put credentials in pyproject.toml, commit .pypirc to Git, or hardcode tokens in CI configuration files. Use environment variables for CI. Use ~/.pypirc (in your home directory, not the project directory) for local publishing. If a token is committed to a repository - even a private one - rotate it immediately. PyPI detects and revokes leaked tokens automatically, but you should not rely on that.

Part 6 - Publishing with Poetry

Poetry integrates building and publishing into a single workflow:

# Configure PyPI token (stored in Poetry's config, not in pyproject.toml)
poetry config pypi-token.pypi pypi-AgEIcGkg...

# Configure TestPyPI (add as a named repository)
poetry config repositories.testpypi https://test.pypi.org/legacy/
poetry config pypi-token.testpypi pypi-xxxx...

# Build + upload to PyPI in one command
poetry publish --build

# Upload to TestPyPI
poetry publish --repository testpypi --build

# Upload pre-built distributions (if you built separately)
poetry publish

Poetry's pypi-token.* configuration is stored in ~/.config/pypoetry/auth.toml (platform-specific), never in pyproject.toml.

Part 7 - The Full Publish Workflow

Part 8 - PyPI Project Page

The project page on PyPI is generated from your pyproject.toml metadata:

[project]
name = "my-package"
version = "1.2.3"
description = "One-line description shown on PyPI search results"
readme = "README.md"       # rendered as long description on project page
license = {text = "MIT"}
authors = [
    {name = "Your Name", email = "[email protected]"},
]
keywords = ["utility", "cli", "data"]

# Classifiers: standardized tags shown on the project page
# Full list: https://pypi.org/classifiers/
classifiers = [
    "Development Status :: 4 - Beta",
    "Intended Audience :: Developers",
    "License :: OSI Approved :: MIT License",
    "Programming Language :: Python :: 3",
    "Programming Language :: Python :: 3.11",
    "Programming Language :: Python :: 3.12",
    "Programming Language :: Python :: 3.13",
    "Topic :: Software Development :: Libraries :: Python Modules",
    "Typing :: Typed",
]

# Links shown in the sidebar of the project page
[project.urls]
Homepage = "https://github.com/yourname/my-package"
Documentation = "https://my-package.readthedocs.io"
Repository = "https://github.com/yourname/my-package"
"Bug Tracker" = "https://github.com/yourname/my-package/issues"
Changelog = "https://github.com/yourname/my-package/blob/main/CHANGELOG.md"

The README.md becomes the long description shown on the project page. PyPI renders Markdown. Always verify rendering on TestPyPI before the real release.

warning

PyPI package names are globally unique and permanent. Once you publish myutils to PyPI, that name is reserved - even if you later delete all releases, the name remains taken. Choose names carefully: use descriptive names with your organization prefix (acme-utils, myco-data-tools) rather than generic names that might conflict with existing packages or prevent future authors from using obvious names.

Part 9 - Private Package Registries

Many organizations need to publish packages for internal use without making them public on PyPI. Private registries provide authentication-gated package hosting.

GitLab Package Registry

GitLab has a built-in package registry for each project and group:

# Publish to GitLab Package Registry using twine
# PROJECT_ID is your GitLab project's numeric ID
twine upload \
  --repository-url https://gitlab.com/api/v4/projects/${PROJECT_ID}/packages/pypi \
  --username gitlab-ci-token \
  --password ${CI_JOB_TOKEN} \
  dist/*

To consume packages from the GitLab registry:

# Install using the registry URL
pip install \
  --index-url https://__token__:${PERSONAL_ACCESS_TOKEN}@gitlab.com/api/v4/projects/${PROJECT_ID}/packages/pypi/simple/ \
  my-internal-package

Configure in pip.conf for persistent use:

# ~/.pip/pip.conf  (or /etc/pip.conf for system-wide)
[global]
extra-index-url = https://__token__:[email protected]/api/v4/projects/123/packages/pypi/simple/

Configure in Poetry:

# Add the GitLab registry as a named source
poetry source add --priority=supplemental gitlab \
  "https://gitlab.com/api/v4/projects/123/packages/pypi/simple/"

# Configure credentials
poetry config http-basic.gitlab __token__ glpat-xxxxx

AWS CodeArtifact

AWS CodeArtifact is a managed package registry that integrates with IAM for authentication:

# Authenticate (token expires after 12 hours)
aws codeartifact login \
  --tool pip \
  --repository my-repo \
  --domain my-domain \
  --domain-owner 123456789012

# After login, pip.conf is automatically updated with the token
pip install my-internal-package

# Publish
aws codeartifact login --tool twine --repository my-repo --domain my-domain
twine upload dist/*

Nexus and Artifactory

Both Nexus Repository Manager and JFrog Artifactory support PyPI-compatible repositories:

# Nexus
twine upload \
  --repository-url https://nexus.example.com/repository/pypi-hosted/ \
  --username $NEXUS_USER --password $NEXUS_PASS \
  dist/*

# Configure pip to use Nexus as an extra source
pip config set global.extra-index-url \
  "https://${NEXUS_USER}:${NEXUS_PASS}@nexus.example.com/repository/pypi-proxy/simple/"

Part 10 - Automating Releases with CI/CD

Manual publish workflows have two failure modes: forgetting steps, and doing them in the wrong order. Automate everything that happens after git push.

GitLab CI - Publish on Version Tag

# .gitlab-ci.yml

stages:
  - test
  - build
  - publish

variables:
  POETRY_VIRTUALENVS_IN_PROJECT: "true"

.python-base:
  image: python:3.12-slim
  before_script:
    - pip install poetry --quiet
    - poetry install --no-interaction --only main,dev

test:
  extends: .python-base
  stage: test
  script:
    - poetry run pytest tests/ --tb=short -q
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_COMMIT_TAG

build-package:
  image: python:3.12-slim
  stage: build
  before_script:
    - pip install build twine --quiet
  script:
    - python -m build
    - twine check dist/*
  artifacts:
    paths:
      - dist/
    expire_in: 1 week
  rules:
    # Only build on version tags: v1.0.0, v2.3.1, etc.
    - if: '$CI_COMMIT_TAG =~ /^v[0-9]+\.[0-9]+\.[0-9]+/'

publish-to-gitlab:
  stage: publish
  image: python:3.12-slim
  needs: [build-package]
  before_script:
    - pip install twine --quiet
  script:
    - twine upload
        --repository-url "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi"
        --username gitlab-ci-token
        --password "${CI_JOB_TOKEN}"
        dist/*
  rules:
    - if: '$CI_COMMIT_TAG =~ /^v[0-9]+\.[0-9]+\.[0-9]+/'

publish-to-pypi:
  stage: publish
  image: python:3.12-slim
  needs: [build-package]
  before_script:
    - pip install twine --quiet
  script:
    - twine upload dist/*
  environment:
    name: pypi
  variables:
    TWINE_USERNAME: __token__
    TWINE_PASSWORD: $PYPI_API_TOKEN   # set in GitLab project CI/CD variables (masked)
  rules:
    - if: '$CI_COMMIT_TAG =~ /^v[0-9]+\.[0-9]+\.[0-9]+/'
      when: manual   # require manual trigger for real PyPI publish

The when: manual on the PyPI publish job requires a human to click "Run" in the GitLab pipeline after inspecting that the TestPyPI upload worked. This is a deliberate safety gate.

GitHub Actions - Trusted Publishing (OIDC)

GitHub Actions supports PyPI's Trusted Publishers feature, which uses OIDC (OpenID Connect) for authentication - no stored API token needed:

# .github/workflows/publish.yml
name: Publish to PyPI

on:
  push:
    tags:
      - "v*"

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install build tools
        run: pip install build twine

      - name: Build distributions
        run: python -m build

      - name: Check distributions
        run: twine check dist/*

      - name: Upload artifacts
        uses: actions/upload-artifact@v4
        with:
          name: dist
          path: dist/

  publish-testpypi:
    needs: build
    runs-on: ubuntu-latest
    environment: testpypi
    permissions:
      id-token: write    # required for OIDC trusted publishing
    steps:
      - uses: actions/download-artifact@v4
        with:
          name: dist
          path: dist/

      - name: Publish to TestPyPI
        uses: pypa/gh-action-pypi-publish@release/v1
        with:
          repository-url: https://test.pypi.org/legacy/

  publish-pypi:
    needs: publish-testpypi
    runs-on: ubuntu-latest
    environment: pypi       # requires manual approval in GitHub environment settings
    permissions:
      id-token: write
    steps:
      - uses: actions/download-artifact@v4
        with:
          name: dist
          path: dist/

      - name: Publish to PyPI
        uses: pypa/gh-action-pypi-publish@release/v1
        # No token needed - OIDC handles authentication

Trusted Publishing requires one-time setup on PyPI: go to your project's settings on pypi.org → "Publishing" → Add a trusted publisher, specifying the GitHub organization, repository, workflow filename, and environment name. After that, no token is needed in the workflow - PyPI verifies the OIDC token issued by GitHub Actions.

This is the most secure publish method: there is no long-lived credential to rotate, steal, or accidentally commit.

Graded Practice

Level 1 - Identify the Wheel

Given these wheel filenames, answer: (a) which Python versions it supports, (b) which platforms it supports, (c) whether it has compiled extensions:

1. pandas-2.2.0-cp312-cp312-win_amd64.whl
2. click-8.1.7-py3-none-any.whl
3. cryptography-42.0.0-cp312-cp312-macosx_14_0_arm64.whl
4. setuptools-69.0.0-py3-none-any.whl
5. numpy-1.26.4-cp311-cp311-manylinux_2_17_x86_64.whl

Show Answer

1. pandas-2.2.0-cp312-cp312-win_amd64.whl

   • (a) CPython 3.12 only - cp312 tag
   • (b) Windows 64-bit only - win_amd64 tag
   • (c) Yes, compiled extensions - cp312 ABI tag indicates C extensions compiled against CPython 3.12 ABI; none ABI would indicate pure Python

2. click-8.1.7-py3-none-any.whl

   • (a) Any Python 3 implementation - py3 tag includes CPython, PyPy, etc.
   • (b) All platforms - any tag
   • (c) No compiled extensions - none ABI tag confirms pure Python

3. cryptography-42.0.0-cp312-cp312-macosx_14_0_arm64.whl

   • (a) CPython 3.12 only - cp312 tag
   • (b) macOS 14.0+ on Apple Silicon (ARM64) only - macosx_14_0_arm64 tag
   • (c) Yes, compiled extensions - cryptography has Rust extensions (the openssl bindings)

4. setuptools-69.0.0-py3-none-any.whl

   • (a) Any Python 3 - py3 tag
   • (b) All platforms - any tag
   • (c) No compiled extensions - pure Python

5. numpy-1.26.4-cp311-cp311-manylinux_2_17_x86_64.whl

   • (a) CPython 3.11 only - cp311 tag
   • (b) Linux x86_64 with glibc >= 2.17 (most modern Linux distributions) - manylinux_2_17_x86_64
   • (c) Yes, compiled extensions - numpy's C/Fortran extensions are what make it fast

Level 2 - Debug This Publish Failure

A developer runs twine upload dist/* and gets:

HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/
File already exists. See https://pypi.org/help/#file-name-reuse for more information.

On inspection:

• pyproject.toml version is 1.2.0
• dist/ contains my-package-1.2.0-py3-none-any.whl and my-package-1.2.0.tar.gz
• The developer says "I just fixed a typo in the README, nothing else changed"

1. What is the error?
2. Why does PyPI reject the upload?
3. What are the two ways to handle this situation?
4. Which is the correct one and why?

Show Answer

1. The error is that version 1.2.0 of my-package has already been uploaded to PyPI. PyPI's file-name-reuse policy prohibits uploading a file with the same name as a previously uploaded file, even if the contents are different.

2. Why PyPI rejects it: PyPI is a permanent, immutable distribution system. Allowing files to be overwritten would break reproducibility - someone who installed my-package==1.2.0 yesterday would get a different package than someone who installs it today, even with an exact pin. This would make version pinning meaningless. The rule is absolute: once a version is uploaded, its files are permanent.

3. Two ways to handle this:

   Option A (incorrect): Delete the release on PyPI and re-upload.

   • PyPI does allow deletion of a release, but it "yank"s it - the version still exists in the history and cannot be reused. You cannot upload a new 1.2.0 after deleting the old one.
   • Even if deletion were clean, users who had 1.2.0 cached or pinned would get different content depending on when they installed.

   Option B (correct): Bump the version and release a new version.

   • Change version = "1.2.0" to version = "1.2.1" in pyproject.toml
   • Rebuild: python -m build
   • Upload: twine upload dist/*
   • Update the CHANGELOG.md: ## [1.2.1] - Fixed README typo in project description

4. Option B is correct. The reason is the SemVer contract and reproducibility. A README-only fix qualifies as a PATCH release (1.2.0 → 1.2.1). The fix is trivially small, but the version still needs to increment. This is not a bureaucratic rule - it is what makes pip install my-package==1.2.0 reliably reproducible forever: it always installs the same bytes, with the typo.

The correct lesson: test your README rendering on TestPyPI before releasing to real PyPI. TestPyPI allows uploading the same version name when testing (with some caveats), and the twine check dist/* command catches many README rendering issues before upload.

Level 3 - Design the Release Pipeline

You are the lead engineer for an internal Python library acme-datatools used by 8 teams at your company. Requirements:

• Hosted on GitLab (self-managed instance at gitlab.acme.com)
• Should be available via pip install acme-datatools from an internal registry - NOT published to public PyPI
• Must run tests on every merge request
• Must publish automatically when a v* tag is pushed to main
• Must NOT be publishable from feature branches or merge requests (even with manual trigger)
• API tokens must not be stored in CI configuration files
• The package must be installable with pip install --index-url https://gitlab.acme.com/.../ acme-datatools

Design the complete solution: the full .gitlab-ci.yml, the authentication mechanism, and the pip.conf configuration to distribute to consuming teams.

Show Answer

Full .gitlab-ci.yml:

# .gitlab-ci.yml for acme-datatools (internal library)

stages:
  - test
  - build
  - publish

variables:
  POETRY_VIRTUALENVS_IN_PROJECT: "true"
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"

.python-base:
  image: python:3.12-slim
  cache:
    key:
      files:
        - poetry.lock
    paths:
      - .venv/
      - .cache/pip/
  before_script:
    - pip install poetry --quiet
    - poetry install --no-interaction --only main,dev

test:
  extends: .python-base
  stage: test
  script:
    - poetry run pytest tests/ --tb=short -q --cov=src --cov-report=xml
    - poetry run ruff check src/ tests/
    - poetry run mypy src/
  coverage: '/TOTAL.+?(\d+\%)/'
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == "main"
    - if: '$CI_COMMIT_TAG =~ /^v[0-9]+\.[0-9]+\.[0-9]+/'

build-package:
  image: python:3.12-slim
  stage: build
  before_script:
    - pip install build twine --quiet
  script:
    - python -m build
    - twine check dist/*
  artifacts:
    name: "$CI_PROJECT_NAME-$CI_COMMIT_TAG"
    paths:
      - dist/
    expire_in: 30 days
  rules:
    # Only build on exact version tags pushed to main
    - if: '$CI_COMMIT_TAG =~ /^v[0-9]+\.[0-9]+\.[0-9]+/ && $CI_COMMIT_REF_NAME == "main"'

publish-internal:
  stage: publish
  image: python:3.12-slim
  needs:
    - job: build-package
      artifacts: true
    - job: test
  before_script:
    - pip install twine --quiet
  script:
    # CI_JOB_TOKEN is automatically provided by GitLab - no stored secret needed
    - twine upload
        --repository-url "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi"
        --username "gitlab-ci-token"
        --password "${CI_JOB_TOKEN}"
        dist/*
  rules:
    # Publish ONLY on version tags \text{---} never on branches or MRs
    - if: '$CI_COMMIT_TAG =~ /^v[0-9]+\.[0-9]+\.[0-9]+/ && $CI_COMMIT_REF_NAME == "main"'

Authentication mechanism:

The CI_JOB_TOKEN is the key. GitLab automatically injects this token into every CI job, scoped to the current project and job. It has permission to write to the project's Package Registry. No stored secret is needed in CI variables for publishing to the same project's registry.

For cross-project publishing (publishing to a group-level registry), use CI_JOB_TOKEN with the group Package Registry API:

script:
  - twine upload
      --repository-url "https://gitlab.acme.com/api/v4/groups/${GROUP_ID}/packages/pypi"
      --username "gitlab-ci-token"
      --password "${CI_JOB_TOKEN}"
      dist/*

pip.conf for consuming teams:

Distribute this configuration to all teams that need to install acme-datatools:

# ~/.pip/pip.conf
# Install with a Personal Access Token (PAT) with read_package_registry scope
# Get your PAT at: https://gitlab.acme.com/-/user_settings/personal_access_tokens

[global]
extra-index-url = https://__token__:${GITLAB_PAT}@gitlab.acme.com/api/v4/projects/123/packages/pypi/simple/

Or add to pyproject.toml of consuming projects (Poetry):

# consuming project's pyproject.toml
[[tool.poetry.source]]
name = "acme-internal"
url = "https://gitlab.acme.com/api/v4/projects/123/packages/pypi/simple/"
priority = "supplemental"

# Configure credentials (done once per developer machine)
poetry config http-basic.acme-internal __token__ glpat-xxxxxxxxxxxx

Why CI_JOB_TOKEN solves the "no stored credentials" requirement:

GitLab's CI_JOB_TOKEN is a short-lived token (valid only for the duration of the job) automatically issued by GitLab to each CI job. It requires no configuration, no rotation, and cannot be leaked via CI variable misconfiguration. The token has exactly the permissions needed (write to this project's Package Registry) and nothing more. This is the principle of least privilege applied to CI authentication.

Key Takeaways

• pip prefers wheels over sdists: always publish both (python -m build produces both). Wheels install instantly (just extract); sdists require a build step.
• Wheel filenames encode compatibility: py3-none-any means pure Python, any platform. cp312-cp312-manylinux_2_17_x86_64 means CPython 3.12, Linux x86_64. Read the filename before debugging why a wheel won't install on your platform.
• Always upload to TestPyPI first: verify metadata rendering, README display, and pip install from the test index before touching real PyPI. PyPI releases are permanent and cannot be overwritten or deleted cleanly.
• Use API tokens, not username/password: scope tokens to the specific project, not "entire account." Use environment variables (TWINE_USERNAME, TWINE_PASSWORD) in CI. Never commit tokens or .pypirc to Git.
• twine check dist/* before every upload: catches metadata errors, README rendering issues, and missing required fields before PyPI rejects your upload.
• PyPI package names are permanent: once myutils is taken (by you or anyone), that name is reserved forever on PyPI. Use organization-prefixed names for internal or niche packages.
• Private registries (GitLab Package Registry, AWS CodeArtifact, Nexus, Artifactory) give you the same pip install experience as PyPI for internal packages. GitLab's CI_JOB_TOKEN provides zero-configuration, zero-secret-storage CI authentication.
• Automate releases with CI on v* tags: never publish manually from a developer laptop if you can avoid it. Tag-triggered CI pipelines are reproducible, auditable, and prevent the "works on my machine" publish failure.
• GitHub Actions Trusted Publishing (OIDC) eliminates stored API tokens entirely for public packages: configure a trusted publisher on PyPI, and the OIDC token from GitHub Actions handles authentication. It is the most secure publish method available.
• Version already exists? Never overwrite - bump the version. A README fix is still a PATCH bump. Every release gets a new version number. This is what makes ==1.2.3 pinning reliable for every user who ever installed it.

What's Next

You have completed Module 05 - Packaging and Environments. You can now:

• Manage dependencies and virtualenvs with Poetry (poetry add, poetry install --only main, poetry lock)
• Interpret and write SemVer version constraints correctly, classify breaking vs non-breaking changes, and read version specifiers in pyproject.toml and requirements.txt
• Build Python distributions with python -m build, validate them with twine check, publish to PyPI and TestPyPI, and configure private registries for internal packages
• Automate the full release pipeline from version tag to PyPI publish in GitLab CI and GitHub Actions

Module 06 - APIs and Web Basics builds directly on this foundation:

• httpx and requests - the packages you now know how to depend on (^2.31), install, and publish will be the subject of Module 06: how to consume and build REST APIs
• FastAPI and Pydantic - the packages you listed as dependencies in pyproject.toml throughout this module become the building blocks of web services in Module 06
• Dependency injection patterns - Module 06 introduces the design patterns behind frameworks like FastAPI; understanding how Poetry groups work (runtime vs dev dependencies) carries forward
• Testing HTTP clients - the pytest skills from Module 04 combine with the package management skills from Module 05 to build testable, publishable API client libraries