Activation Functions Compared - Interactive Visualization

Activation functions are the nonlinearities that let neural networks learn functions beyond simple linear maps. Without them, stacking layers is equivalent to a single matrix multiply. This visualization plots each function and its derivative side by side so you can immediately see where gradients vanish (sigmoid/tanh flat regions), where dead neurons occur (ReLU at negative values), and why GELU has become the default in transformers.

• Compare curves and derivatives for ReLU, sigmoid, tanh, GELU, Swish, and Leaky ReLU on the same plot
• Understand the vanishing gradient problem: sigmoid and tanh derivatives approach zero for large inputs, starving earlier layers
• See why ReLU revolutionized deep learning in 2012 - near-constant gradient for positive inputs allows much deeper networks
• Learn about dead neurons: once a ReLU input is negative, the gradient is zero and the neuron never recovers
• Understand GELU and Swish: smooth approximations to ReLU that outperform it in transformers and modern architectures

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.