Python Internals Module Projects - Engineering Challenges

Level: Intermediate → Engineering | Format: Build, not follow

These are not exercises. They are not walkthroughs. They are project specifications.

You receive a requirements document, a starter skeleton, and expected outputs. You build the implementation. The same way you would at work.

Each project is designed so that the Python internals concepts from the module are not optional - they are structurally required. You cannot complete Project 01 without understanding bytecode, the dis module, and how CPython compiles source to code objects. You cannot complete Project 02 without understanding sys.setprofile, the call event model, and tracemalloc's snapshot API. The projects force the concepts into place through real engineering constraints.

How to Approach These Projects

Read the full spec before writing a line of code. The requirements are ordered for comprehension, not for implementation. For internals projects especially, the design decisions - how to model the value stack, how to build a call tree rather than a flat list, where to attach metadata - must be resolved before implementation begins.

Sketch the data model first. Before writing any code for Project 01, sketch what a single disassembled instruction looks like as a data structure. Before writing any code for Project 02, sketch what a call tree node looks like: what fields does it need to compute own time vs total time? The data model determines everything downstream.

Build incrementally and test as you go. Implement one method at a time. Because these tools operate on deterministic inputs (source code strings, known functions), testing is straightforward - run the same input and check the output against your expectations without mocks or fakes.

The hints section exists for when you are genuinely stuck - not as a first resort. Try to work through the problem for at least 20 minutes before reading a hint.

Extension challenges are serious. If you complete the core requirements quickly, the extensions will push you into territory that connects directly to production tooling - coverage measurement, compiler optimisation, and continuous profiling.

The Two Projects

Project 01 - Bytecode Visualizer

Core internals concepts tested: ast module, compile(), dis.get_instructions(), code objects, the CPython value stack model, bytecode instruction categories.

You build a bytecode analysis tool that takes Python source code as a string and exposes four analysis methods: show_ast() pretty-prints the abstract syntax tree with type annotations, show_bytecode() produces a colour-annotated disassembly with human-readable explanations for each instruction, show_stack_trace() simulates the CPython value stack step by step so you can see the exact state of the stack after every instruction executes, and compare() accepts a second source string and produces a side-by-side bytecode diff. The tool also counts instructions by category and can export the full analysis as JSON. This project forces you to read the CPython bytecode documentation carefully - there is no shortcut.

Estimated time: 5–7 hours for core requirements.

Project 02 - Mini Profiler Tool

Core internals concepts tested: sys.setprofile(), call and return event handling, call tree construction, own time vs total time, tracemalloc, context managers, decorator protocols.

You build a production-quality profiler that works as both a context manager (with Profiler() as p:) and a decorator (@p.profile). Time profiling uses sys.setprofile() to intercept call and return events and builds a call tree - not a flat list - so own time can be computed correctly. Memory profiling uses tracemalloc to record peak usage, total allocations, and the top allocation sites by traceback. The profiler prints a formatted report table using only the standard library, supports comparing two runs side by side, and can export JSON for CI threshold enforcement. This project is a direct simplified re-implementation of what cProfile does under the hood.

Estimated time: 6–8 hours for core requirements.

Skills Map

Concept	Project 01	Project 02
ast module and AST traversal	Yes
compile() and code objects	Yes
dis.get_instructions()	Yes
CPython value stack model	Yes
Bytecode instruction categories	Yes
sys.setprofile()		Yes
Call event handling		Yes
Call tree construction		Yes
Own time vs total time		Yes
tracemalloc		Yes
Context manager protocol		Yes
Decorator protocol		Yes
JSON export	Yes	Yes
Stdlib-only formatted tables		Yes

Setup

All projects are self-contained. No third-party libraries required beyond the Python standard library.

# Create a working directory
mkdir internals-projects
cd internals-projects

# Run your implementation with
python visualizer.py
python profiler.py

Python 3.10 or later is assumed. dis.get_instructions(), tracemalloc, sys.setprofile, and ast.dump() with indent support are all standard library. No external dependencies.

What Good Looks Like

A complete submission for either of these projects satisfies four criteria:

1. All expected outputs match exactly - the sample outputs in each spec are exact. Your implementation must reproduce them character for character, except for values that vary by timing or memory allocation size.
2. Internals APIs are used correctly - dis.get_instructions() is used rather than parsing the output of dis.dis(). tracemalloc.take_snapshot() is used rather than measuring before-and-after sys.getsizeof. sys.setprofile() is called and cleared correctly so the profiler does not leak state.
3. The data model is explicit - the call tree in Project 02 is a real tree, not a stack hack. The instruction analysis in Project 01 is a list of structured objects, not raw strings being formatted on the fly.
4. The code could be extended - someone reading your implementation could add a new instruction category to the visualizer without touching the display code, or add a new report section to the profiler without touching the data collection code.

These are the same criteria a senior engineer would apply in a code review.