The 4 Production Deployment Tiers

Tier 1

Shadow Mode

→

Tier 2

Canary

→

Tier 3

HITL

→

Tier 4

Autonomous

Click a tier above to see full details

🚀

Select a deployment tier above to see when to use it, how to implement it, and how to know when to move up or down.

Shadow Mode

Agent runs in parallel — outputs logged, never acted upon

Risk: None

When to Use

Shadow mode is your first deployment for any new agent. Run it from day one, before you have any confidence in the agent's behavior. Use it whenever the task involves real data, irreversible side effects, or any consequence you can't easily undo. The agent observes the same inputs as your existing human or system process — but outputs go to a log, not to production.

Implementation Pattern

Fork the production request to both the existing process and the agent. The existing process executes normally. The agent's output is captured and logged alongside the ground truth from the existing process. You now have pairs of (agent_output, ground_truth) that you can analyze to build your eval suite and understand where the agent diverges from expected behavior.

Monitoring Requirements

Log everything: agent output, latency, token count, and cost per run. Compare agent output to ground truth on a sample basis — start with 10–20% manual review, then build automated evals once you understand the pattern of divergences. Track divergence rate over time. A rising divergence rate suggests model drift or distribution shift in your inputs.

Maturity Requirement

Run shadow mode for at least 2 weeks before considering promotion to canary. You want 500–1000 shadow runs covering a representative sample of real inputs, including edge cases. If you cannot identify a clear set of cases where the agent diverges from ground truth, your sample is too small or too homogeneous.

Example Tasks at This Tier

Data pipeline validation Document summarization Customer query classification Code review suggestions Report generation

📈

Escalation Path: Promote to Canary When... Divergence rate from ground truth is below your threshold (e.g., under 5% on your key metrics) for at least 2 weeks of shadow runs. Your eval suite passes consistently. You have identified and documented the edge cases where the agent fails and understand their frequency. Move down (or stay in shadow): if divergence is high, rising, or concentrated in high-value cases.

Canary Deployment

Agent handles N% of real traffic — limited blast radius

Risk: Low

When to Use

Canary is appropriate after 2+ weeks of successful shadow mode. The agent has demonstrated acceptable divergence from ground truth on a representative sample. You are ready to let the agent's outputs have real effects — but limited ones. Start at 1–5% of traffic, with a clear process to increase or roll back based on outcome metrics.

Implementation Pattern

Route N% of production requests to the agent (via feature flag, percentage routing, or a/b test framework). The remaining (100-N)% goes to your existing process. Compare outcomes between the two groups — not just output quality, but downstream business metrics: error rates, customer satisfaction scores, rework rates, cost per resolved issue.

Monitoring Requirements

Real-time dashboard comparing agent traffic vs control traffic on your key metrics. Automated alerts if agent error rate exceeds 2x the control group. Cost monitoring per agent run — canary traffic is real spend. Latency comparison — if the agent is significantly slower than the existing process, users in the canary bucket notice. Daily review of agent runs for qualitative assessment.

Traffic Ramp Schedule

Start at 1%. Hold for 3 days with no incidents. Increase to 5%. Hold for 1 week. Increase to 10%. Hold for 2 weeks. If metrics remain clean, proceed to 25%, then 50%. Each step requires explicit sign-off from your team. Never increase traffic while investigating an incident. Roll back to the previous percentage instantly if key metrics degrade.

Example Tasks at This Tier

Email draft generation Support ticket routing Product description writing Low-value API calls Internal tooling recommendations

📈

Escalation Path: Promote to HITL When... The agent has handled at least 1000 real production requests with no critical failures. Business metrics in the canary group match or exceed the control group. Cost per run is within budget. You are ready to let the agent take high-stakes actions — but you want a human backstop for irreversible ones. Roll back to shadow: on any critical incident, or if business metrics fall 10%+ below control group for 48 hours.

Human-in-the-Loop

Agent acts — pauses for human approval before irreversible actions

Risk: Medium

When to Use

HITL is the right tier for agents that need to take consequential, potentially irreversible actions: sending emails, committing code, making payments, modifying production data, interacting with external systems. The agent is proven enough to do the work — but not yet trusted to act without a human seeing the proposed action first. This is the appropriate steady-state for most enterprise deployments.

Implementation Pattern

Use LangGraph's interrupt_before mechanism on high-risk tool nodes. When the agent reaches an irreversible action, the graph pauses, serializes state, and returns control to the caller. A human reviewer sees the proposed action and approves or rejects. The agent resumes from the checkpoint with the human decision injected into state. Build a review UI — Slack bot, web app, or email — that surfaces pending actions clearly.

Monitoring Requirements

Track: interrupt rate (how often the agent hits a HITL checkpoint per run), approval rate (what % of proposed actions humans approve), rejection reasons (categorize why humans reject, use this to improve the agent), time-to-review (SLA for human reviewers to act), and escalation rate (how often the agent reaches a state where a human must take over entirely rather than just approve one step).

Reviewer Experience Design

The HITL interface is part of your product. Show the reviewer: the original user goal, the agent's reasoning up to this point, the proposed action with arguments clearly displayed, the estimated impact, and a one-click approve/reject. Never show raw JSON to non-technical reviewers. Add a "request more context" option so the reviewer can ask the agent for justification before deciding.

Example Tasks at This Tier

Customer communication agents Code review and PR creation Financial reporting Data pipeline orchestration IT automation with change risk

📈

Escalation Path: Promote to Autonomous When... For a specific action class (e.g., "send internal Slack summary"), approval rate exceeds 98% over 500+ reviews with zero critical rejections. Reviewers consistently approve within seconds without any additional context request — indicating they find the action obviously correct. Move action-class-by-action-class, never promote the entire agent at once. Roll back: on any rejected action that caused harm, or if approval rate drops below 95% for any action class.

Fully Autonomous

Agent acts without human review — earned on high-confidence tasks only

Risk: High (manage carefully)

When to Use

Full autonomy is earned, not assumed. It applies to specific action classes where the agent has a demonstrated track record: 98%+ approval rate over hundreds of HITL reviews, zero critical failures over 30+ days, and a reversibility plan in place. Do not grant autonomy to an action class because it "seems safe" — grant it because the data shows it is. Most production agents should remain partially in HITL indefinitely.

Implementation Pattern

Remove the interrupt for proven action classes while keeping it for others. Implement async review: the agent acts immediately, but a human reviewer sees a log of all autonomous actions taken in the last hour and can raise an alert or trigger a rollback. This is not HITL (no pre-approval) but it is not blind (post-hoc review). Add automatic rollback triggers: if anomaly detection fires on an autonomous run, pause autonomy for that action class and route to HITL until investigated.

Monitoring Requirements

The most rigorous monitoring tier. Required: real-time cost alerting (autonomous agents can burn budget fast), output distribution anomaly detection (statistical deviation from baseline triggers HITL escalation), latency p99 monitoring, tool call frequency alerting (spike in tool calls per run = potential loop), error rate dashboards with < 5-minute lag, and weekly adversarial red-team testing against your running agent.

Rollback Capability

You must be able to disable an autonomous agent and revert its effects in under 5 minutes. This means: feature flags that cut off the agent immediately, audit logs that record every action with enough detail to reverse it, a runbook that anyone on your on-call rotation can execute without the agent's author present, and tested rollback procedures (run a drill quarterly). If you cannot roll back, you cannot safely ship autonomy.

Example Tasks That Can Earn This Tier

Internal log summarization Low-risk data enrichment Read-only analytics queries Draft generation (no send) Monitoring alert triage

⚠️

Escalation Path: Roll Back to HITL When... Any autonomous action causes unexpected harm or data corruption. Cost anomaly: autonomous runs cost more than 3x the expected budget for 2+ consecutive runs. Output anomaly: statistical distribution of outputs deviates significantly from baseline (your anomaly detector fires). Model version changes: always return to HITL after a model update until the new version earns the autonomous track record from scratch.

www.engineersofai.com · AI Letters #09 · Agentic AI A-Z Series