Feed-Forward Network (FFN) Sub-layer - Interactive Visualization

The feed-forward sub-layer in a transformer block is a two-layer MLP applied independently to each token position. It expands token representations from d_model to d_ff (typically 4×) using a nonlinear activation like GELU or SwiGLU, then projects back down. This expansion-contraction pattern stores factual knowledge and accounts for roughly two-thirds of all transformer parameters.

• Step-through animation of the FFN forward pass: Linear1 → activation → Linear2
• Configurable d_model (256–2048) and expansion ratio (1×–8×) with live parameter count
• Compare GELU, ReLU, and SwiGLU activations - including SwiGLU gate mechanism used in LLaMA
• Parameter breakdown bars showing how FFN dominates total model size

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.