:::tip 🎮 Interactive Playground Visualize this concept: Try the Data Drift Detection demo on the EngineersOfAI Playground - no code required. :::

Monte Carlo and Observability Platforms

The 6-Month Build vs. 2-Day Deploy

The team had spent six months building their own data observability system. Three engineers. Twelve monitoring checks covering freshness, volume, and a handful of schema checks. Three Slack integration dashboards. The system covered 15% of their warehouse tables - specifically, the 15% of tables that had experienced quality issues before. They were proud of it.

Then Monte Carlo ran a proof of concept. In two days, with zero configuration changes to the data warehouse, Monte Carlo covered 100% of their tables. It automatically learned the historical behavior of every table - row count patterns, update frequencies, column distributions - and set anomaly detection baselines from the data itself. Within the first week, it surfaced three data quality issues the team had never checked for. One of them had been silently degrading a recommendation model for 3 months.

The team's build covered the known unknowns - the issues they had previously experienced. Monte Carlo covered the unknown unknowns - the issues no one had thought to check for. Both teams built something defensible. But they built fundamentally different things.

This is the central tension in the data observability platform market: you can build a checks-based system that covers what you know to look for, or you can deploy a ML-based platform that covers what you have not yet thought to check for. The right answer depends on your team size, budget, and maturity - and increasingly, the answer is "you need some of both."

Why Observability Platforms Exist

Rule-based data quality checks - Great Expectations assertions, dbt tests, SQL assertions - are powerful and precise. You define a rule, the system checks it, and you get a pass/fail signal. The problem is that rule-based checks scale with the number of checks you write, not with the number of tables you have.

An organization with 500 tables and 3 data engineers cannot write and maintain meaningful quality checks for every column of every table. They will cover their 20 most critical tables well and leave the rest unmonitored. The 481 unmonitored tables contain unknown unknowns - quality issues that will only be discovered when they affect something downstream.

Observability platforms solve this coverage problem with two approaches:

1. ML-based anomaly detection (Monte Carlo, Anomalo): automatically profile every table, learn normal behavior from historical data, and alert when current behavior deviates from the learned baseline. Zero configuration required to cover the entire warehouse.

2. Threshold-based automated rule generation (Bigeye): analyze historical data to automatically suggest and set thresholds for each column, removing the manual threshold-tuning burden.

Both approaches address the same problem from different angles. Neither replaces explicit business logic checks (revenue must be positive, user count must be monotonically increasing) - those still require human definition. But both dramatically expand monitoring coverage without requiring a proportional increase in engineering time.

Monte Carlo: How It Works

Monte Carlo is the market leader in data observability as of 2025. The architecture is notable for what it does not require: Monte Carlo never moves your data or reads the data values themselves. It works exclusively from query logs and metadata.

What Monte Carlo reads:

• Query logs (Snowflake, Redshift, BigQuery, Databricks query history)
• Information schema metadata (table schemas, sizes, update times)
• Query results of statistical monitoring queries it runs against your warehouse

What Monte Carlo never reads:

• Actual data values in your tables
• PII or sensitive column contents
• Source system data outside the warehouse

This metadata-first approach has two advantages: it is faster than reading full table data, and it satisfies strict data security requirements without requiring sensitive data to leave the warehouse.

The core ML engine works as follows. For each table, Monte Carlo computes time-series metrics: daily row count, last update time, null rates per column, value distribution statistics. It trains a time-series anomaly detection model on the historical values of each metric. When the current value deviates significantly from the predicted range, it fires an alert.

Monte Carlo Circuit Breaker

One of Monte Carlo's most operationally valuable features is the circuit breaker: the ability to automatically halt downstream pipeline execution when a quality anomaly is detected upstream.

When Monte Carlo detects that staging.events has a volume anomaly, it can send a webhook to Airflow that sets the downstream dbt_run_mart_layer task to FAILED before it executes. This prevents the anomaly from propagating to the mart layer, the BI tools, and the ML feature store.

The circuit breaker works via the Monte Carlo API:

# Airflow DAG: check Monte Carlo before running downstream tasks

from airflow.decorators import task, dag
from datetime import datetime
import requests
import os

MONTE_CARLO_API = "https://api.getmontecarlo.com/graphql"
MC_API_KEY = os.environ["MONTE_CARLO_API_KEY"]

@task
def check_monte_carlo_health(tables: list) -> bool:
    """
    Query Monte Carlo for active incidents on upstream tables.
    Returns True if safe to proceed, False if incidents are open.
    """
    query = """
    query GetTableIncidents($tableFullNames: [String!]!) {
        getTableIncidents(tableFullNames: $tableFullNames, status: ACTIVE) {
            incidents {
                id
                type
                severity
                table { fullTableId }
                createdTime
            }
        }
    }
    """
    headers = {
        "Authorization": f"Bearer {MC_API_KEY}",
        "Content-Type": "application/json",
    }
    payload = {
        "query": query,
        "variables": {"tableFullNames": tables},
    }

    resp = requests.post(MONTE_CARLO_API, json=payload, headers=headers)
    resp.raise_for_status()

    incidents = resp.json()["data"]["getTableIncidents"]["incidents"]
    critical = [i for i in incidents if i["severity"] in ("SEV_1", "SEV_2")]

    if critical:
        incident_ids = [i["id"] for i in critical]
        raise ValueError(
            f"Monte Carlo has {len(critical)} active critical incidents on upstream tables. "
            f"Incident IDs: {incident_ids}. "
            f"Halting pipeline until incidents are resolved."
        )

    return True

@dag(schedule_interval="0 6 * * *", start_date=datetime(2024, 1, 1))
def mart_refresh_pipeline():
    health_check = check_monte_carlo_health(
        tables=["prod_db.staging.events", "prod_db.staging.users"]
    )
    # ... downstream tasks only run if health_check passes

The Platform Landscape

Bigeye: Threshold-Based Monitoring

Bigeye (acquired by Cloudera in 2024) takes a different approach from Monte Carlo. Instead of ML-based anomaly detection, Bigeye uses automated threshold generation: it analyzes historical data for each column and automatically sets alert thresholds based on observed distributions.

Bigeye's strength is interpretability. Every alert has a clear, human-readable threshold: "row count dropped below 85,000 rows" rather than "row count is outside the ML model's predicted range." This makes alerts easier to understand, adjust, and trust.

Bigeye is better suited for teams that want explicit, auditable thresholds and are willing to tune them over time. Monte Carlo is better suited for teams that want zero-configuration coverage and are comfortable with ML-based signals.

Anomalo: ML-Based for Enterprise

Anomalo is Monte Carlo's closest ML-based competitor. Key differentiators: stronger support for unstructured data quality, tighter integration with Databricks, and a focus on root cause explanation (not just "this is anomalous" but "here is the likely root cause").

Soda Core: Open-Source, Check-Based

Soda Core is the open-source data quality framework. It is check-based (not ML-based), meaning you define checks in YAML and Soda runs them. This is the right tool when you have known quality rules to enforce and want open-source with no SaaS fees.

# soda-checks/mart_user_features.yml
checks for mart.user_features:
  - row_count > 100000:
      name: "Minimum row count check"
      fail: error

  - freshness(created_at) < 2h:
      name: "Freshness SLA - must update within 2 hours"
      fail: error

  - missing_count(user_id) = 0:
      name: "user_id must never be null"
      fail: error

  - duplicate_count(user_id) = 0:
      name: "No duplicate user IDs"
      fail: error

  - invalid_percent(churn_score) < 1%:
      name: "churn_score must be in [0, 1]"
      valid range: [0, 1]
      fail: error

  - avg(ltv_90d) between 50 and 500:
      name: "LTV 90d average sanity check"
      warn: true
      fail: error

Running Soda in Airflow:

from airflow.operators.python import PythonOperator
from soda.scan import Scan

def run_soda_checks(
    checks_file: str,
    data_source_name: str,
    scan_definition: str,
) -> None:
    """Run Soda Core checks from an Airflow task."""
    scan = Scan()
    scan.set_verbose(True)
    scan.set_scan_definition_name(scan_definition)
    scan.set_data_source_name(data_source_name)

    # Load data source configuration
    scan.add_configuration_yaml_file("soda/configuration.yml")

    # Load checks
    scan.add_sodacl_yaml_file(checks_file)

    exit_code = scan.execute()

    if exit_code == 2:  # Critical errors
        raise ValueError(f"Soda critical checks failed for {checks_file}. Halting pipeline.")
    elif exit_code == 1:  # Warnings
        # Log warnings but allow pipeline to continue
        import logging
        logging.warning(f"Soda warnings for {checks_file}. Review before deployment.")

    scan.assert_no_error_nor_warning()

Datafold: Diff-Based, dbt-Native

Datafold takes a fundamentally different approach. Instead of monitoring a table's health over time, Datafold compares two versions of the same data - before and after a code change.

The primary use case is dbt CI: when a developer opens a PR that changes a dbt model, Datafold runs both the current production version and the PR version, then shows you a row-level diff. You can see exactly which rows changed, which column distributions shifted, and what percentage of the output was affected by the code change.

This is not traditional observability - it is regression testing for data transformations. It catches bugs before they reach production rather than after.

# Example: integrating Datafold into a GitLab CI pipeline
# .gitlab-ci.yml (relevant excerpt)

# Run in GitLab CI on every MR to the dbt repository
datafold_diff:
  stage: quality
  script:
    # Run the modified dbt models in a dev environment
    - dbt run --target ci --select state:modified+
    # Run Datafold diff between CI and production
    - |
      datafold dbt \
        --ci-config-id $DATAFOLD_CI_CONFIG_ID \
        --run-type pull_request \
        --target-name ci \
        --commit-sha $CI_COMMIT_SHA \
        --base-branch $CI_DEFAULT_BRANCH
  artifacts:
    reports:
      # Datafold posts a comment to the MR with the diff summary

dbt-Native Observability: dbt-expectations + elementary

For teams already using dbt, the most frictionless observability approach combines:

1. dbt test: built-in tests for nullability, uniqueness, referential integrity, accepted values
2. dbt-expectations: 50+ additional test types (statistical, distribution, pattern-matching)
3. elementary: open-source dbt package that generates a full observability dashboard from dbt test results

# models/mart/user_features.yml - dbt-expectations tests

version: 2
models:
  - name: user_features
    description: "User-level feature table for ML models. Updated hourly."
    config:
      meta:
        owner: ml-platform
        freshness_sla_hours: 2
    columns:
      - name: user_id
        tests:
          - not_null
          - unique
          - dbt_expectations.expect_column_values_to_be_of_type:
              column_type: bigint

      - name: churn_score
        tests:
          - not_null
          - dbt_expectations.expect_column_values_to_be_between:
              min_value: 0
              max_value: 1
          - dbt_expectations.expect_column_mean_to_be_between:
              min_value: 0.05
              max_value: 0.50
              # Alert if mean churn score moves outside this range
          - dbt_expectations.expect_column_stdev_to_be_between:
              min_value: 0.05
              max_value: 0.30

      - name: ltv_90d
        tests:
          - dbt_expectations.expect_column_values_to_be_between:
              min_value: 0
              max_value: 50000
          - dbt_expectations.expect_column_quantile_values_to_be_between:
              quantile: 0.95
              min_value: 50
              max_value: 2000

    tests:
      # Table-level: row count should not drop by more than 10% from yesterday
      - dbt_expectations.expect_table_row_count_to_be_between:
          min_value: 100000
          max_value: 50000000

Elementary adds an observability layer on top of dbt tests:

-- elementary generates this observability view automatically
-- from running test results stored in the elementary schema

SELECT
    model_name,
    test_name,
    status,
    execution_time,
    detected_at,
    full_refresh
FROM elementary.test_results
WHERE detected_at >= CURRENT_TIMESTAMP - INTERVAL '7 days'
ORDER BY detected_at DESC

Build vs. Buy: The Decision Framework

Factor	Build Custom	Soda Core (OSS)	Monte Carlo / Anomalo
Table count	less than 50	50–500	200+
Team size	1–2 engineers	2–5 engineers	5+ engineers
Annual cost	~$0 infra	~$0 + setup time	$50k–$200k/yr
Coverage	What you write	What you write	Entire warehouse
Unknown unknowns	Not covered	Not covered	Covered
Integration depth	Custom	Good (Airflow, Slack)	Best in class
Setup time	Weeks to months	Days	Hours to days

The decision is primarily about coverage vs. cost. For a startup with 50 tables and 2 data engineers, Soda Core is the right answer. For a scale-up with 500 tables and a dedicated data platform team, Monte Carlo's automated coverage of unknown unknowns pays for itself in the first incident it catches that manual checks would have missed.

The hybrid approach that works at most organizations: use dbt tests + Soda Core for known business rules (these are explicit, auditable, cheap), and layer Monte Carlo on top for unknown anomaly detection. The dbt tests catch "revenue is negative" and "user_id is null." Monte Carlo catches "the distribution of this column shifted in a way nobody anticipated."

:::danger Alert fatigue is the silent killer of observability programs An observability system that sends 50 alerts per day and 40 of them are false positives is worse than no observability system. Engineers learn to ignore all alerts. The real incidents get lost in the noise. Start with a high threshold (wide error bands, Z-score greater than 3) and tighten thresholds only after the system has demonstrated reliability. Track your false positive rate. If it is above 20%, loosen thresholds rather than letting engineers develop alert fatigue. :::

:::warning Monte Carlo and similar platforms are not a replacement for dbt tests ML-based anomaly detection catches behavioral anomalies. It does not enforce business rules. A model that learns "revenue can be between $100 and$1M per day" will not catch a bug that introduces a negative revenue transaction - because it has never seen negative revenue and will alert. But a dbt test that asserts revenue_usd > 0 will catch it immediately. Business logic checks and anomaly detection are complementary, not substitutable. :::

Integration Patterns: Connecting Observability to the Stack

The value of an observability platform is multiplied when it is integrated into every relevant part of the data stack:

Airflow integration: observability platform sends a webhook to Airflow when a critical incident is detected. The webhook sets all pending downstream tasks to FAILED or SKIPPED, preventing bad data from propagating. This is the circuit breaker pattern.

Slack integration: each table or pipeline should have a dedicated #data-alerts-{team} channel. Route all incidents for that team's tables to that channel. Avoid routing all alerts to a shared channel - alert fatigue and ownership ambiguity both increase.

PagerDuty integration: critical incidents (impacting ML models or executive reporting) should trigger PagerDuty. Use a severity threshold - only incidents classified as P1 or P2 (by an explicit classification rule or by Monte Carlo's severity scoring) should page on-call engineers.

Jira integration: automatically create Jira tickets for all incidents, pre-populated with: affected table, incident type, severity, detection time, and a link to the observability platform for investigation. Assign to the technical owner of the table.

Interview Q&A

Q: What is Monte Carlo and how does it detect data anomalies?

A: Monte Carlo is a data observability platform that automatically monitors all five pillars of data observability (freshness, volume, schema, distribution, lineage) across your entire data warehouse without requiring manual check configuration. It works by reading query logs and metadata - not the data values themselves - to compute time-series metrics for every table (row counts, update times, null rates, distribution statistics). It trains ML anomaly detection models on each metric's historical values and alerts when current values deviate significantly from the predicted range. The key advantage over manual checks is coverage: Monte Carlo covers 100% of your tables automatically, including tables you never thought to write checks for.

Q: When would you choose Soda Core over Monte Carlo?

A: Soda Core is the right choice when: your data stack has fewer than 200 tables, you have a small team that can maintain explicit check configurations, your monitoring needs are primarily rule-based rather than anomaly-based (known business rules vs. unknown drift detection), and you have strict cost constraints. Soda Core is also the right choice when you need explicit, auditable thresholds that stakeholders can review and approve - ML-based anomaly detection is harder to audit because the threshold is derived from a model, not a human-specified rule. Monte Carlo is the right choice when you need coverage of unknown unknowns at scale, when you have a large table estate that would require disproportionate engineering time to monitor with explicit checks, or when the cost of a single missed data incident exceeds the platform cost.

Q: What is Datafold and how does it differ from traditional observability?

A: Datafold is a data diff tool that compares two versions of a dataset - typically before and after a code change. It is used primarily in dbt CI to catch regressions before they reach production: when a developer modifies a dbt model, Datafold runs both the current production version and the modified version, then shows a row-level diff highlighting which rows changed and how column distributions shifted. This is regression testing for data transformations, not runtime anomaly detection. Traditional observability watches production data for unexpected changes over time. Datafold catches changes before deployment. For a mature data organization, you want both: Datafold in CI to prevent regressions, and Monte Carlo or Soda in production to catch unexpected runtime behavior.

Q: How do you integrate observability alerting into an Airflow pipeline?

A: The key integration is the circuit breaker pattern: the observability platform monitors upstream tables, and when it detects a critical anomaly, it sends a signal to Airflow to halt the downstream pipeline. In practice, this is implemented as an Airflow sensor or API check task at the beginning of each DAG. The task queries the observability platform API for active incidents on the upstream tables it depends on. If critical incidents exist, the task fails, which halts all downstream tasks. This prevents bad data from propagating from a broken source table through the transformation layers to the mart, BI, and ML layers. Without this integration, a volume anomaly in staging.events would silently produce a wrong mart.user_features table, which would produce wrong ML predictions before anyone noticed.

Q: Explain the build vs. buy tradeoff for data observability tooling.

A: The build vs. buy decision in data observability comes down to three factors: scale, coverage, and known vs. unknown failure modes. For small-scale deployments (under 200 tables), building custom monitoring with SQL checks stored in PostgreSQL and visualized in Grafana is entirely practical and costs almost nothing beyond engineering time. For medium-scale (200–500 tables), Soda Core provides a structured framework for check-based monitoring without a SaaS subscription. For large-scale deployments (500+ tables, especially with ML workloads), commercial platforms like Monte Carlo provide coverage of unknown unknowns - the anomalies nobody thought to write checks for. The economic argument for commercial platforms is that the engineering time to build equivalent coverage (automated anomaly detection across all tables) typically exceeds the SaaS cost within the first year, especially accounting for the ongoing maintenance burden of a custom system.

Q: What is the elementary dbt package and what observability capabilities does it add?

A: Elementary is an open-source dbt package that extends dbt's test framework into a full observability system. It works by storing dbt test results in a dedicated elementary schema in your warehouse and then providing: a data observability dashboard (rendered from those test results), anomaly detection for column-level metrics (row count, null rate, distribution) that extends dbt's built-in tests, and a test result history so you can see how quality metrics have trended over time. It generates an HTML observability report that shows the health of every model over the past days or weeks. Elementary is the right choice for teams that are already heavily invested in dbt and want observability without adding another SaaS platform. It does not cover unknown unknowns as comprehensively as Monte Carlo's ML-based approach, but it provides good coverage for known checks at zero incremental licensing cost.