EngineersofAI LogoCoursesBlogResearch LabAI LettersThe LabCode BankInteractive 3DKodrEarnestJobs
Skip to main content
Interactive 3D/Policy Gradient (REINFORCE)
Status
Episode: 0
Ep reward: 0
Mean(10): 0
Entropy: 1
Learning rate1.0e-2
Baseline
Subtract mean reward to reduce variance
Train
REINFORCE: update ∇log π(a|s)·G. Blue=P(Left), Green=P(Right). Goal: keep pole upright (reward=steps).

Policy Gradient (REINFORCE) - Interactive Visualization

Policy gradient methods optimize a policy directly by gradient ascent on expected return. REINFORCE computes the gradient as the log probability of each action weighted by the return received - actions that led to high returns get their probabilities increased. Unlike Q-learning, policy gradient methods can handle continuous action spaces and directly optimize what we care about: total reward.

  • Watch the policy network update after each episode: good actions become more probable, bad actions less probable
  • See the REINFORCE gradient: nabla_theta J = E[nabla_theta log pi(a|s) * G_t] - log probability weighted by return
  • Understand high variance: REINFORCE uses full episode returns, making gradient estimates noisy - baselines reduce this
  • Add a baseline (average return) and watch variance drop - the agent learns faster with the same sample count
  • Compare REINFORCE vs Q-learning: policy gradient is on-policy and requires fresh episodes; Q-learning can reuse experience

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.