Module 01 - Metaprogramming

Reading time: ~12 minutes | Level: Advanced

Here is a question that separates framework users from framework builders:

class Field:
    def __set_name__(self, owner, name):
        self.name = name

    def __set__(self, obj, value):
        obj.__dict__[self.name] = value

    def __get__(self, obj, objtype=None):
        if obj is None:
            return self
        return obj.__dict__.get(self.name)


class ModelMeta(type):
    def __new__(mcs, name, bases, namespace):
        fields = {k: v for k, v in namespace.items() if isinstance(v, Field)}
        cls = super().__new__(mcs, name, bases, namespace)
        cls._fields = fields
        return cls


class Model(metaclass=ModelMeta):
    def __init_subclass__(cls, table_name=None, **kwargs):
        super().__init_subclass__(**kwargs)
        cls._table_name = table_name or cls.__name__.lower()


class User(Model, table_name="users"):
    name = Field()
    email = Field()

How many separate metaprogramming hooks are firing in that code?

The answer is four. __set_name__ is called on each Field when the class body finishes. ModelMeta.__new__ intercepts class creation to collect fields. __init_subclass__ fires on User because it subclasses Model. And every attribute access on an instance will go through __get__ and __set__ on the descriptors.

If you cannot trace each hook, when it fires, and why it exists -- you cannot build frameworks. You can only use them.

This module closes that gap.

Why Metaprogramming Matters

Every major Python framework is built on metaprogramming. Not as a novelty. As a necessity.

• Django uses ModelBase (a metaclass) to collect field definitions, build database schemas, and generate querysets -- all from a class declaration.
• SQLAlchemy uses DeclarativeMeta to map class attributes to table columns, intercept attribute access for SQL generation, and manage session-level identity maps.
• Pydantic uses __set_name__ and descriptors to build validation pipelines at class definition time, not at runtime.
• dataclasses uses code generation with exec and compile to synthesise __init__, __repr__, __eq__, and __hash__ from field declarations.
• pytest uses import hooks to rewrite assert statements into detailed introspection calls before your test code even executes.

These are not edge cases. They are the foundational patterns of Python's ecosystem. If you want to build systems at this level -- or even debug them effectively -- you need to understand what is happening beneath the class statement.

note

This module assumes you have completed the Intermediate course. You should already be comfortable with classes, inheritance, MRO, decorators, closures, and the basics of how CPython works. We are going deeper now -- into the machinery that creates classes themselves.

What This Module Covers

Metaprogramming in Python is the practice of writing code that manipulates code. Not string manipulation or eval tricks. Python has a formal, well-designed set of hooks that let you intercept and modify class creation, attribute access, and even the import process itself.

By the end of this module, you will be able to:

• Build metaclasses that validate, register, and transform classes at definition time
• Implement the descriptor protocol to control attribute access at the object level
• Use __init_subclass__ as the modern, lightweight alternative to metaclasses
• Understand how __set_name__ gives descriptors automatic awareness of their host class
• Create classes dynamically using type() and code generation
• Write custom import hooks that transform how Python loads modules

Module Topics

01 -- Metaclasses

type is not just a function that tells you an object's type. It is the metaclass -- the class of all classes. When you write class Foo:, Python calls type('Foo', (object,), {...}) to build it.

You will understand how type.__new__ allocates the class object, how type.__init__ initialises it, and how type.__call__ orchestrates instance creation when you call Foo(). You will write custom metaclasses that intercept all three phases.

But more importantly, you will understand when metaclasses are the right tool. Django's ModelBase collects field definitions from the class namespace, validates them, builds _meta options, and registers the model with the application registry -- all before a single instance exists. SQLAlchemy's DeclarativeMeta does similar work to map Python classes to database tables. These are metaclasses earning their complexity.

You will also understand metaclass inheritance -- how a metaclass propagates through a class hierarchy, what happens when two base classes have different metaclasses, and how to resolve metaclass conflicts.

Key concepts: type, __new__, __init__, __call__ on metaclasses, __prepare__, metaclass conflict resolution, class registries, ModelBase, DeclarativeMeta

02 -- Descriptors

Descriptors are the single most important protocol in Python's data model. property, classmethod, staticmethod, bound methods, __slots__ -- all of them are implemented through the descriptor protocol.

A descriptor is any object that defines __get__, __set__, or __delete__. When Python looks up an attribute, it checks whether the attribute found in the class is a descriptor, and if so, calls the appropriate protocol method instead of returning the object directly.

You will understand the critical distinction between data descriptors (which define __set__ or __delete__) and non-data descriptors (which only define __get__). This distinction controls attribute lookup priority: data descriptors override instance __dict__, while non-data descriptors do not. This is why property can intercept writes but a regular method can be shadowed by an instance attribute.

You will build production-quality validators -- type-checked fields, range-validated numbers, string constraints -- using the descriptor protocol. And you will understand why frameworks like Django and Pydantic use descriptors as the backbone of their field systems.

Key concepts: __get__, __set__, __delete__, data vs non-data descriptors, attribute lookup order, property internals, classmethod internals, validator descriptors

03 -- __init_subclass__

Metaclasses are powerful but heavy. For many common patterns -- subclass registration, validation, default configuration -- __init_subclass__ is the correct tool.

Added in Python 3.6, __init_subclass__ is a hook on the parent class that fires whenever a new subclass is defined. It receives the subclass and any keyword arguments from the class statement. No metaclass needed. No import complications. No metaclass conflicts.

You will build a plugin registration system where subclasses automatically register themselves with a central registry. You will implement validation that rejects invalid subclass definitions at class creation time -- not at instantiation, not at runtime, but at the moment the class statement executes.

You will also understand the trade-offs. __init_subclass__ cannot modify the class namespace before the class is created (metaclass __new__ can). It cannot change the class creation process itself. It is a post-creation hook, and knowing that boundary is essential.

Key concepts: __init_subclass__, keyword arguments in class statements, subclass registries, plugin patterns, validation at definition time, comparison with metaclasses

04 -- __set_name__

When you write name = Field() inside a class body, how does the Field instance know it has been assigned to the attribute name? Before Python 3.6, it could not -- you had to pass the name explicitly, as in name = Field('name'), a notorious source of bugs and redundancy.

__set_name__ solves this. After the class body executes, Python calls __set_name__(self, owner, name) on every descriptor in the class namespace. The descriptor receives both the owning class and the attribute name it was assigned to.

This is the hook that makes Pydantic's Field(), Django's model fields, and SQLAlchemy's Column() work without requiring you to repeat the attribute name. You will implement it and understand its interaction with the descriptor protocol and metaclasses.

Key concepts: __set_name__, descriptor naming, owner class reference, interaction with metaclasses, Pydantic Field, Django Field

05 -- Dynamic Class Creation

Every class statement in Python is syntactic sugar for a call to type(). The three-argument form -- type(name, bases, namespace) -- creates a new class at runtime with an arbitrary name, base classes, and attributes.

This is not a curiosity. It is how Python's standard library builds classes programmatically. collections.namedtuple generates a class from a string template using exec. @dataclass generates method source code, compiles it, and injects it into the class namespace. typing.NamedTuple constructs classes dynamically with type annotations.

You will understand the full class creation pipeline: namespace preparation via __prepare__, body execution, type.__new__, and __init_subclass__. You will use type() directly, generate code with exec and compile, and understand the security and debugging implications of code generation.

You will also build a simple DSL (domain-specific language) -- a declarative interface that generates Python classes from a higher-level description. This is the pattern behind every ORM, serialisation library, and configuration system in Python.

Key concepts: type() three-argument form, exec, compile, __prepare__, namedtuple internals, @dataclass code generation, DSL construction

06 -- Import Hooks and the Import System

Python's import system is not a black box. It is a fully extensible protocol built on finders and loaders, exposed through importlib and sys.meta_path.

When you write import foo, Python walks sys.meta_path looking for a finder that claims responsibility for foo. The finder returns a loader. The loader creates the module, executes its code, and installs it in sys.modules. Every step is hookable.

You will write custom importers that load modules from non-standard sources -- databases, remote URLs, encrypted archives. You will implement lazy imports that defer module loading until first attribute access. You will understand how pytest rewrites assert statements by installing an import hook that transforms the AST before compilation.

You will also understand import-time side effects: why importing a module can execute arbitrary code, how circular imports break, and why if __name__ == '__main__' exists.

Key concepts: importlib, sys.meta_path, finders and loaders, ModuleSpec, lazy imports, AST transformation, import-time side effects, circular imports

Module Architecture

The topics in this module form a layered system. Each layer builds on the one below it.

Descriptors are the foundation -- they control attribute access and are used everywhere. __set_name__ extends descriptors with automatic naming. Metaclasses control class creation itself and depend on understanding descriptors. __init_subclass__ is a modern alternative to metaclasses for common patterns. Dynamic class creation is the underlying mechanism that metaclasses orchestrate. Import hooks operate at the module level, intercepting code before classes are even defined.

Module Projects

Project	Core Skills
Custom ORM Core	Metaclasses, descriptors, __set_name__, dynamic SQL generation
Plugin Framework	__init_subclass__, class registries, import hooks, lazy loading

Custom ORM Core

Build a mini SQLAlchemy-style ORM from scratch. You will implement a Model metaclass that collects Field descriptors, generates CREATE TABLE SQL, and provides save(), load(), and filter() methods. The fields will use __set_name__ for automatic column naming and the descriptor protocol for type validation. By the end, you will have a working ORM that maps Python classes to SQLite tables -- and you will understand exactly how Django and SQLAlchemy do the same thing at a much larger scale.

Plugin Framework

Build an auto-discovering plugin system. Plugins are classes that subclass a Plugin base, and __init_subclass__ automatically registers them in a central registry. The framework uses import hooks to discover and load plugin modules from a configurable directory. It supports dependency ordering, enable/disable toggling, and lifecycle hooks (on_load, on_unload). This is the pattern behind pytest plugins, Flask extensions, and Celery task discovery.

Prerequisites

• Python Intermediate course complete (or equivalent depth)
• Comfortable with classes, inheritance, MRO, and super()
• Understand decorators and closures at the implementation level
• Familiar with how CPython resolves attribute lookup (__dict__, class hierarchy)
• Familiar with basic __new__ and __init__ on regular classes

tip

If type(MyClass) returning <class 'type'> does not immediately make sense to you, review the Intermediate module on Python Internals before starting here. Metaprogramming builds directly on the object model.

How to Use This Module

The lessons are ordered deliberately. Read them in sequence.

Descriptors (02) and __set_name__ (04) are the foundation. Metaclasses (01) and __init_subclass__ (03) build on that understanding. Dynamic Class Creation (05) ties the mechanisms together. Import Hooks (06) extends the reach to the module level.

The projects integrate multiple topics. Do not attempt them until you have completed the lessons.

danger

Metaprogramming is powerful and easy to misuse. The goal of this module is not to make you use metaclasses everywhere. It is to make you understand when they are necessary, when a simpler hook suffices, and when plain classes are the right choice. The best metaprogramming is invisible to the user of your API.

The Engineering Standard

This module is written for engineers who build the tools that other engineers use.

Every concept connects to a real framework, a real design decision, or a real production system. You will not just learn what __init_subclass__ does -- you will understand why it was added to the language, what problem it solved, and when it replaced metaclasses in CPython's own standard library.

By the end of this module, you will read Django's ModelBase, SQLAlchemy's DeclarativeMeta, or Pydantic's ModelMetaclass and understand every line. Not because you memorised an API. Because you understand the machinery.

That is the difference between a developer who uses frameworks and an engineer who can build them.