EngineersofAI LogoCoursesBlogResearch LabAI LettersThe LabCode BankInteractive 3DKodrEarnestJobs
Skip to main content
Interactive 3D/Content Moderation ML System Architecture
Moderation Pipeline
1
Content Submission0ms
Text / image / video upload
2
Hash Matching<1ms
PhotoDNA, exact match known-bad content. Handles ~40% of obvious violations.
3
ML Classifier (Text)50ms
Fine-tuned BERT / Llama classifier
4
Confidence Threshold<1ms
Auto-approve if conf < 20% | Auto-reject if conf > 85% | Else → human queue
5
Human Review Queue2-48h
Queue depth: 5,000 items. Reviewers see context + ML score.
6
Final Decision-
Approve / Remove / Partial (age-gate, blur, label)
7
Model Feedbackasync
Reviewer decisions → training dataset → periodic retraining
Category - FP vs FN Cost
False Positive (remove good content)
Silenced legitimate speech
False Negative (miss bad content)
Real harm to users
Precision vs Recall Trade-off
Auto-reject threshold →0.00.51.00.50.95Precision (avoid FP)Recall (catch bad)
Routing Breakdown
Auto-approve20%
Human review65%
Auto-reject15%
Human review queue: ~5,000 items
Controls
Auto-approve if conf <20%
Auto-reject if conf >85%
Review queue depth5,000
Options
Hash matching: PhotoDNA compares image hashes against known CSAM/terrorist content database. Near 0ms, 100% recall on known content.

Human review: 60-120 reviews/hour per reviewer. Traumatic at scale - major platforms rotate reviewers and provide counseling.

Widening the band reduces auto-routing but increases human cost.

Content Moderation ML System Architecture - Interactive Visualization

Content moderation at platform scale requires a multi-stage pipeline that balances speed, accuracy, and cost. The first layer uses cryptographic hash matching (PhotoDNA) to instantly block known bad content without any ML inference. The second layer runs an ML classifier (fine-tuned BERT for text, ResNet for images) to score unknown content. High-confidence predictions auto-enforce; borderline predictions route to human review queues. The fundamental tension is false positive rate (removing legitimate content) versus false negative rate (allowing harmful content), with asymmetric costs per category. Widening the confidence band reduces automation but shrinks human review cost.

  • Hash matching: PhotoDNA compares image perceptual hashes against NCMEC database - <1ms, 100% recall on known content
  • ML classifier: fine-tuned BERT (text) or ResNet (image) scores content 0-1 for each violation category
  • Auto-routing: confidence < 20% → auto-approve; confidence > 85% → auto-remove; between → human review queue
  • Human review: 60-120 items per hour per reviewer - queue depth determines staffing and review latency
  • Appeals system: creator appeals go to senior reviewers, overrides become high-quality training labels
  • False positive cost varies by category: removing news coverage of violence (FP) vs leaving actual violence (FN)

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.