Courses Blog Research Lab AI Letters The Lab Code Bank Interactive 3DKodr Earnest Jobs

Interactive 3D/AI Safety Evaluations - Benchmark Heatmap

Safety Benchmark Heatmap - Threshold: 70%

Model	HarmBench harmlessness	TruthfulQA truthfulness	WMDP security	CyberSec security	ToxiGen toxicity	BBQ harmlessness	Avg
GPT-4	82	59 below threshold	78	71	85	90	77.5
Claude-3	91	68 below threshold	92	88	93	95	87.8
Llama-3	74	51 below threshold	65 below threshold	60 below threshold	77	81	68.0
Gemini	87	62 below threshold	84	79	88	92	82.0

Score Distribution by Model

GPT-4

77.5%

Claude-3

87.8%

Llama-3

68.0%

Gemini

82.0%

Controls

Models

GPT-4Claude-3Llama-3Gemini

Category Filter

all

harmlessness

truthfulness

security

toxicity

Threshold

Score70%

Sort By

Safety evals measure different dimensions: harmlessness (refuse harmful requests), truthfulness (accurate outputs), security (resist misuse), toxicity (avoid harmful language).

AI Safety Evaluations - Benchmark Heatmap - Interactive Visualization

AI safety evaluation benchmarks measure different dimensions of model safety and alignment. HarmBench tests refusal of harmful requests across attack types. TruthfulQA measures accuracy on questions where humans tend to be misled. WMDP (Weapons of Mass Destruction Proxy) tests resistance to dangerous knowledge. ToxiGen evaluates toxic content generation. BBQ tests social bias. No single model dominates all categories - there are real tradeoffs.

HarmBench: tests refusal rate against 400+ harmful request categories and jailbreak attacks
TruthfulQA: 817 questions where humans give false answers - tests calibrated truthfulness
WMDP: biosecurity/cybersecurity/chemistry harmful knowledge - model should NOT know this
ToxiGen: 274k synthetic toxic statements - model should not generate or agree with them
Safety vs capability tradeoff: over-refusing (false positives) also costs helpfulness

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.