Student Performance Prediction

Reading time: ~40 min · Interview relevance: High · Target roles: ML Engineer, Data Scientist, EdTech Engineer

Opening: The Student You Could Have Helped

A university analytics team built an early warning system in 2017. The system predicted, with 81% accuracy by week 6 of a 16-week semester, which students would receive a D or F in their introductory calculus course. The model ingested LMS login events, assignment submission patterns, quiz scores, and discussion participation.

The team ran a retrospective analysis on the previous three years of data. The model would have flagged 73% of students who ultimately failed - at week 6, with 10 weeks remaining. They presented this to department leadership. Leadership was impressed. They approved deployment.

The deployed system identified at-risk students and sent their names to their assigned academic advisor. Advisors received a list each Monday. By semester 3, advisors reported that they were overwhelmed with the volume of flags and lacked a systematic protocol for outreach. Some students never heard from their advisor. Others were contacted but had no clear next step. The intervention loop - flag, outreach, support - was never closed. The system predicted accurately and intervened ineffectively.

This is the central challenge of student performance prediction in production. The ML problem - predicting who is at risk - is technically solvable with standard gradient boosting methods. The harder problem is the causal chain from prediction to intervention to outcome. A prediction that does not trigger an effective intervention does not improve student outcomes. It may actually harm them: students flagged as "at risk" in systems with low administrative capacity may be labeled without receiving support, while the labeling itself can have stigmatizing effects.

This lesson covers the technical components: feature engineering from LMS clickstream data, dropout prediction models, grade prediction, survival analysis for time-to-dropout, fairness auditing, FERPA compliance, and - critically - the intervention design that determines whether predictions have any value.

---

Why This Exists: The Scale of Student Dropout

The dropout problem in education is substantial. At US four-year institutions, roughly 40% of students who begin a bachelor's degree do not complete it within six years. At community colleges, the rate is higher. MOOC completion rates are notoriously low - typically 5-15% of enrollees complete a course.

The economic cost is enormous: uncompleted degrees represent lost earning potential, loan obligations without the degree that would service them, and institutional costs. The human cost is harder to quantify but real: students who drop out often do so after a crisis period during which earlier intervention might have been effective.

The ML opportunity is that modern LMSs (Canvas, Moodle, Blackboard, edX) generate rich behavioral logs. Every login, video view, assignment download, quiz attempt, discussion post, and help request is timestamped and attributable to a student. From this data, models can detect early warning signals that human advisors, managing hundreds of students simultaneously, would miss.

The caveat is that most studies of early warning systems measure predictive accuracy, not intervention effectiveness. A system can predict with 85% accuracy that a student will drop out and still not prevent a single dropout if the prediction is not connected to an effective intervention.

---

Historical Context: From Grade Books to Predictive Analytics

Pre-2010 - Grade-Based Early Alerts: Most institutions had "early alert" systems where faculty submitted at-risk reports manually when students missed assignments or performed poorly on early assessments. These were reactive, required faculty attention, and had low coverage.

2012 - KDDCUP 2015: The KDD Cup 2015 challenge released a large dropout prediction dataset from XuetangX (a major Chinese MOOC platform) with 120,542 students and 39 courses. This dataset benchmarked ML approaches to dropout prediction and showed that gradient boosting on LMS behavioral features outperformed human judgment.

2013-2015 - Purdue Signals, EAB Navigate, Civitas Learning: Commercial early warning products launched, selling predictive analytics as a service to universities. These products demonstrated market demand but also revealed the intervention design gap.

2018-2022 - FERPA and Algorithmic Accountability: The Department of Education issued guidance on FERPA compliance for predictive analytics. Scholars like Audrey Watters published influential critiques of educational data mining's privacy and fairness implications. Institutions began demanding bias audits for student prediction systems.

2023+ - Causal Inference Focus: The research literature shifted from prediction accuracy to causal effects of interventions. RCT studies (random assignment to receive early warning outreach) showed mixed results - some interventions worked, many did not, and a few had negative effects (students who were contacted and did not improve felt more labeled and anxious).

---

Core Concepts

Feature Engineering from LMS Clickstream

The raw data from an LMS is a stream of events: student $s$ performed action $a$ at time $t$ on resource $r$. From this stream, features need to be computed that capture behavioral patterns predictive of performance.

Engagement frequency features:

• Logins per week (normalized by course average)
• Days since last login
• Assignment submission rate (submitted / due)
• Video completion rate
• Discussion post count and reply count

Engagement recency features:

• Days since any LMS activity
• Streak: consecutive days with LMS activity
• Last week's activity vs baseline weeks

Performance trajectory features:

• Mean quiz score
• Quiz score trend (slope of score over time)
• Assignment late submission rate
• Grade trajectory: recent grades vs early grades

Temporal pattern features:

• Time-of-day activity distribution (regular schedule vs irregular)
• Session duration statistics
• Binge vs distributed activity (studying 12 hours before a deadline vs daily study)

Social features:

• Discussion participation (posts, replies, views)
• Peer interaction count in group assignments

The key challenge: missing data. Students who are disengaging stop generating events, meaning absence of data is itself a signal. Impute missing engagement features with zeros (absence = no activity) rather than mean imputation, because zero truly reflects the student's behavior.

Dropout Prediction Models

Dropout is typically framed as a binary classification problem at a prediction point $t^*$ (e.g., end of week 3): given features from weeks 1 to $t^*$, predict whether the student will drop the course.

Logistic regression is the baseline. Interpretable, fast, requires minimal data. Coefficients are directly actionable: "each missed assignment increases log-odds of dropout by 0.6."

Gradient boosting (LightGBM, XGBoost) is the standard in production. Handles nonlinear interactions and missing values natively, and achieves top performance on tabular behavioral data. The KDDCUP 2015 winning solution used gradient boosting.

Neural networks have been explored but rarely outperform gradient boosting on the feature-engineered tabular datasets typical of LMS data. They are useful when integrating sequential data (raw clickstream sequences) that are hard to compress into aggregate features.

Key evaluation metrics:

• AUC-ROC: standard discrimination metric. A model predicting at random achieves 0.5; a perfect model achieves 1.0. Models on the KDDCUP 2015 dataset achieve 0.85-0.90 AUC.
• Precision at early cutoff: If you can only contact 50 students per advisor per week, what is the precision of the top-50 flagged students? This is operationally more relevant than overall AUC.
• Recall of high-risk subgroups: Does the model catch at-risk students in all demographic subgroups at similar rates?

Survival Analysis for Time-to-Dropout

Binary dropout prediction answers "will this student drop?" Survival analysis answers "when will this student drop?" This is more useful for intervention timing.

The hazard function $h(t)$ is the instantaneous dropout rate at time $t$ given survival to time $t$:

$h(t) = \lim_{\Delta t \to 0} \frac{P(t \leq T < t + \Delta t | T \geq t)}{\Delta t}$

The survival function $S(t) = P(T > t)$ gives the probability of remaining enrolled beyond time $t$.

Cox Proportional Hazards Model:

$h(t | \mathbf{x}) = h_0(t) \exp(\boldsymbol{\beta}^T \mathbf{x})$

The baseline hazard $h_0(t)$ is nonparametric (estimated from data); the covariate vector $\mathbf{x}$ multiplies it via an exponential. This allows the model to handle time-varying covariates (engagement metrics updated weekly) and right-censored observations (students who have not yet dropped out by the analysis date).

The partial likelihood for Cox regression:

$\mathcal{L}(\boldsymbol{\beta}) = \prod_{i: \text{dropped}} \frac{\exp(\boldsymbol{\beta}^T \mathbf{x}_i)}{\sum_{j \in R(t_i)} \exp(\boldsymbol{\beta}^T \mathbf{x}_j)}$

where $R(t_i)$ is the risk set at dropout time $t_i$ - students who are still enrolled.

Survival analysis is particularly useful for adaptive intervention timing: flag students whose predicted hazard rate is accelerating, not just those with high cumulative dropout probability.

Grade Prediction

Grade prediction at week $k$ of a semester answers: given what we know so far, what final grade will this student receive? This drives resource recommendations ("students at this grade trajectory benefit from office hours") and serves as a leading indicator for at-risk flags.

Regression approaches: predict the continuous final grade from features up to week $k$. Linear regression as a baseline, gradient boosting as the production model.

The early-semester challenge: at week 1, you have almost no data. Feature importance shifts over the semester. An ensemble that mixes prior probability (grade distribution for this student's profile) with current performance evidence (weighted more as more data accumulates) often outperforms pure ML models in the first few weeks.

Fairness in Student Prediction

Predictive models for student performance can exhibit demographic disparities. Protected attributes in education include race, gender, first-generation student status, disability status, and Pell grant eligibility (a proxy for socioeconomic status).

Three fairness criteria commonly applied to student prediction:

Demographic parity: $P(\hat{Y} = 1 | A = 0) = P(\hat{Y} = 1 | A = 1)$ for protected attribute $A$. The flagging rate should be equal across groups. Problem: if there are true differences in outcome rates across groups due to structural factors, demographic parity forces the model to flag low-risk students from disadvantaged groups.

Equal opportunity: $P(\hat{Y} = 1 | Y = 1, A = 0) = P(\hat{Y} = 1 | Y = 1, A = 1)$. The model should catch students who will drop out at equal rates across groups. This means no group is systematically missed.

Predictive parity (calibration): $P(Y = 1 | \hat{Y} = 1, A = 0) = P(Y = 1 | \hat{Y} = 1, A = 1)$. When the model flags a student, the probability they actually are at risk should be equal across groups. This means the model's risk scores mean the same thing regardless of group membership.

These criteria are mutually incompatible in general (Chouldechova, 2017). Which to prioritize depends on the intervention design. If intervention resources are limited, equal opportunity ensures disadvantaged groups are not systematically under-served. If there are costs to false positives (e.g., stigmatization, advisor time), predictive parity ensures the flags that consume resources are equally valid across groups.

FERPA Compliance

The Family Educational Rights and Privacy Act (FERPA) regulates educational records for students at US institutions receiving federal funding. Key implications for ML systems:

• Student records (including model-derived risk scores) are protected educational records if they are "directly related to a student and maintained by an educational agency."
• Sharing records outside the institution requires written consent, with exceptions for "school officials with legitimate educational interest."
• Students have the right to inspect and request amendment of their educational records.
• Risk scores generated by ML models may be FERPA-protected records.

Practical compliance steps: data minimization (collect and retain only what is needed), purpose limitation (data collected for advising may not be repurposed for other uses), access controls (risk scores visible only to advisors with educational interest), retention limits (delete records when no longer needed), and a process for student record access requests.

---

Mermaid Diagram: Student Performance Prediction Pipeline

---

Code Examples

Engagement Feature Engineering from Clickstream

import pandas as pd
import numpy as np
from typing import List, Dict

def engineer_engagement_features(
    events_df: pd.DataFrame,
    courses_df: pd.DataFrame,
    prediction_week: int,
    course_id: str
) -> pd.DataFrame:
    """
    Engineer student engagement features from LMS event logs.

    Args:
        events_df: DataFrame with columns [student_id, event_type,
                   resource_id, timestamp, course_id]
        courses_df: DataFrame with course info [course_id, start_date, end_date]
        prediction_week: week number to predict dropout from (1-indexed)
        course_id: course to compute features for

    Returns:
        DataFrame with one row per student and engagement feature columns
    """
    course = courses_df[courses_df['course_id'] == course_id].iloc[0]
    start_date = pd.Timestamp(course['start_date'])

    # Filter to this course and up to prediction week
    df = events_df[events_df['course_id'] == course_id].copy()
    df['timestamp'] = pd.to_datetime(df['timestamp'])
    df['week'] = ((df['timestamp'] - start_date).dt.days // 7) + 1
    df = df[df['week'] <= prediction_week]

    # Per-student feature computation
    features = []

    for student_id, student_df in df.groupby('student_id'):
        f = {'student_id': student_id}

        # Total activity
        f['total_events'] = len(student_df)
        f['active_days'] = student_df['timestamp'].dt.date.nunique()
        f['active_weeks'] = student_df['week'].nunique()

        # Event type breakdown
        event_counts = student_df['event_type'].value_counts()
        f['login_count'] = event_counts.get('login', 0)
        f['quiz_attempt_count'] = event_counts.get('quiz_attempt', 0)
        f['assignment_submit_count'] = event_counts.get('assignment_submit', 0)
        f['video_view_count'] = event_counts.get('video_view', 0)
        f['discussion_post_count'] = event_counts.get('discussion_post', 0)
        f['resource_access_count'] = event_counts.get('resource_access', 0)

        # Recency: days since last event (as of prediction week end)
        prediction_cutoff = start_date + pd.Timedelta(weeks=prediction_week)
        last_event = student_df['timestamp'].max()
        f['days_since_last_activity'] = (prediction_cutoff - last_event).days

        # Regularity: weekly activity consistency
        weekly_events = student_df.groupby('week').size()
        # Pad missing weeks with 0
        all_weeks = pd.Series(0, index=range(1, prediction_week + 1))
        weekly_events = all_weeks.add(weekly_events, fill_value=0)
        f['weekly_activity_mean'] = weekly_events.mean()
        f['weekly_activity_std'] = weekly_events.std()
        f['weeks_with_zero_activity'] = (weekly_events == 0).sum()

        # Trend: is engagement increasing or decreasing?
        if len(weekly_events) >= 3:
            x = np.arange(len(weekly_events))
            slope = np.polyfit(x, weekly_events.values, 1)[0]
            f['activity_trend_slope'] = slope
        else:
            f['activity_trend_slope'] = 0.0

        # Last week vs first week activity
        if prediction_week >= 2:
            last_week_activity = weekly_events.get(prediction_week, 0)
            first_week_activity = weekly_events.get(1, 0)
            f['last_vs_first_week_ratio'] = (
                last_week_activity / (first_week_activity + 1)
            )
        else:
            f['last_vs_first_week_ratio'] = 1.0

        # Session behavior: time-of-day irregularity
        if len(student_df) > 1:
            hours = student_df['timestamp'].dt.hour.values
            f['activity_hour_std'] = np.std(hours)  # High std = irregular schedule
        else:
            f['activity_hour_std'] = 0.0

        features.append(f)

    return pd.DataFrame(features).fillna(0)

Dropout Prediction with LightGBM and SHAP

import lightgbm as lgb
import shap
import numpy as np
import pandas as pd
from sklearn.model_selection import StratifiedKFold
from sklearn.metrics import roc_auc_score, precision_recall_curve
from typing import Dict, List, Tuple

def train_dropout_predictor(
    features_df: pd.DataFrame,
    labels: pd.Series,
    feature_cols: List[str],
    n_folds: int = 5
) -> Tuple[lgb.LGBMClassifier, Dict]:
    """
    Train dropout prediction model with cross-validation.

    Args:
        features_df: engagement features per student
        labels: binary dropout labels (1=dropped out)
        feature_cols: list of feature column names to use
        n_folds: number of CV folds

    Returns:
        trained model, evaluation metrics dict
    """
    X = features_df[feature_cols].values
    y = labels.values

    lgb_params = {
        'objective': 'binary',
        'metric': 'auc',
        'learning_rate': 0.05,
        'num_leaves': 31,
        'min_child_samples': 20,
        'feature_fraction': 0.8,
        'bagging_fraction': 0.8,
        'bagging_freq': 5,
        'reg_alpha': 0.1,
        'reg_lambda': 0.1,
        'verbose': -1
    }

    cv = StratifiedKFold(n_splits=n_folds, shuffle=True, random_state=42)
    oof_preds = np.zeros(len(y))
    fold_aucs = []

    for fold, (train_idx, val_idx) in enumerate(cv.split(X, y)):
        X_train, X_val = X[train_idx], X[val_idx]
        y_train, y_val = y[train_idx], y[val_idx]

        model = lgb.LGBMClassifier(**lgb_params, n_estimators=500)
        model.fit(
            X_train, y_train,
            eval_set=[(X_val, y_val)],
            callbacks=[lgb.early_stopping(50, verbose=False)]
        )

        oof_preds[val_idx] = model.predict_proba(X_val)[:, 1]
        fold_auc = roc_auc_score(y_val, oof_preds[val_idx])
        fold_aucs.append(fold_auc)
        print(f"Fold {fold+1}: AUC = {fold_auc:.4f}")

    # Train final model on all data
    final_model = lgb.LGBMClassifier(**lgb_params, n_estimators=500)
    final_model.fit(X, y)

    oof_auc = roc_auc_score(y, oof_preds)
    print(f"\nOverall OOF AUC: {oof_auc:.4f}")
    print(f"CV AUC: {np.mean(fold_aucs):.4f} +/- {np.std(fold_aucs):.4f}")

    metrics = {
        'oof_auc': oof_auc,
        'cv_auc_mean': float(np.mean(fold_aucs)),
        'cv_auc_std': float(np.std(fold_aucs))
    }

    return final_model, metrics


def explain_dropout_prediction(
    model: lgb.LGBMClassifier,
    student_features: np.ndarray,
    feature_names: List[str],
    top_k: int = 5
) -> Dict:
    """
    Generate SHAP-based explanation for a student's dropout risk.
    Returns top contributing factors for advisor communication.
    """
    explainer = shap.TreeExplainer(model)
    shap_values = explainer.shap_values(student_features)

    # For binary classification, shap_values is a list of 2 arrays
    # Take the positive class SHAP values
    if isinstance(shap_values, list):
        sv = shap_values[1]  # Class 1 (dropout)
    else:
        sv = shap_values

    # Get top contributing features
    feature_importance = list(zip(feature_names, sv[0]))
    feature_importance.sort(key=lambda x: abs(x[1]), reverse=True)

    risk_factors = []
    protective_factors = []

    for feature, shap_val in feature_importance[:top_k]:
        explanation = {
            'feature': feature,
            'shap_value': float(shap_val),
            'direction': 'increases risk' if shap_val > 0 else 'reduces risk'
        }
        if shap_val > 0:
            risk_factors.append(explanation)
        else:
            protective_factors.append(explanation)

    # Human-readable summary
    risk_factor_summary = []
    feature_descriptions = {
        'days_since_last_activity': 'has not logged in recently',
        'weeks_with_zero_activity': 'has had multiple inactive weeks',
        'activity_trend_slope': 'engagement is declining over time',
        'assignment_submit_count': 'has submitted few assignments',
        'quiz_attempt_count': 'has made few quiz attempts',
        'discussion_post_count': 'has not participated in discussions'
    }

    for factor in risk_factors:
        description = feature_descriptions.get(
            factor['feature'],
            factor['feature'].replace('_', ' ')
        )
        risk_factor_summary.append(description)

    return {
        'risk_factors': risk_factors,
        'protective_factors': protective_factors,
        'risk_factor_summary': risk_factor_summary,
        'top_risk_factor': risk_factors[0]['feature'] if risk_factors else None
    }

Survival Analysis with lifelines

from lifelines import CoxPHFitter, KaplanMeierFitter
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

def fit_dropout_survival_model(
    survival_df: pd.DataFrame,
    duration_col: str = 'weeks_to_dropout',
    event_col: str = 'dropped_out',
    covariate_cols: list = None
) -> CoxPHFitter:
    """
    Fit Cox Proportional Hazards model for dropout timing.

    Args:
        survival_df: DataFrame with duration (weeks to dropout or end of semester),
                     event (True if dropped), and covariate features
        duration_col: column name for duration
        event_col: column name for event indicator
        covariate_cols: feature columns to include in the model

    Returns:
        fitted CoxPHFitter
    """
    if covariate_cols is None:
        covariate_cols = [c for c in survival_df.columns
                         if c not in [duration_col, event_col]]

    cph = CoxPHFitter()
    cph.fit(
        survival_df[[duration_col, event_col] + covariate_cols],
        duration_col=duration_col,
        event_col=event_col,
        show_progress=False
    )

    print(cph.summary[['coef', 'exp(coef)', 'p']].to_string())
    print(f"\nConcordance Index: {cph.concordance_index_:.4f}")

    return cph


def predict_dropout_risk_timeline(
    model: CoxPHFitter,
    student_features: pd.DataFrame,
    weeks: list = None
) -> pd.DataFrame:
    """
    Predict dropout probability at each future week for a student.
    Useful for prioritizing intervention timing.
    """
    if weeks is None:
        weeks = list(range(1, 17))  # 16-week semester

    survival_functions = model.predict_survival_function(student_features)
    dropout_probs = 1 - survival_functions

    results = pd.DataFrame(
        dropout_probs.T.values,
        index=student_features.index,
        columns=[f'P(dropout by week {w})' for w in survival_functions.index]
    )

    return results


def compute_hazard_acceleration(
    model: CoxPHFitter,
    student_features: pd.DataFrame,
    current_week: int
) -> float:
    """
    Compute whether the student's hazard rate is accelerating.
    High positive acceleration -> intervene now.
    """
    sf = model.predict_survival_function(student_features)
    if current_week < 2 or current_week + 1 > len(sf):
        return 0.0

    # Discrete hazard at current and previous week
    h_current = (sf.iloc[current_week - 1] - sf.iloc[current_week]) / sf.iloc[current_week - 1]
    h_prev = (sf.iloc[current_week - 2] - sf.iloc[current_week - 1]) / max(sf.iloc[current_week - 2], 1e-9)

    return float(h_current.mean() - h_prev.mean())

Fairness Audit with Demographic Parity

import pandas as pd
import numpy as np
from scipy import stats
from typing import Dict, List

def audit_prediction_fairness(
    predictions_df: pd.DataFrame,
    score_col: str = 'risk_score',
    threshold: float = 0.5,
    outcome_col: str = 'actual_dropout',
    protected_cols: List[str] = None
) -> Dict:
    """
    Comprehensive fairness audit for student dropout predictions.

    Computes:
    - Demographic parity: equal flagging rates across groups
    - Equal opportunity: equal TPR across groups
    - Predictive parity: equal precision across groups
    - Disparate impact ratio (4/5ths rule)

    Args:
        predictions_df: DataFrame with scores, outcomes, and protected attributes
        score_col: risk score column (0 to 1)
        threshold: decision threshold for flagging
        outcome_col: actual dropout indicator
        protected_cols: demographic columns to audit

    Returns:
        dict with fairness metrics per protected attribute
    """
    if protected_cols is None:
        protected_cols = [c for c in predictions_df.columns
                         if c not in [score_col, outcome_col, 'student_id']]

    df = predictions_df.copy()
    df['predicted'] = (df[score_col] >= threshold).astype(int)

    results = {}

    for col in protected_cols:
        groups = df[col].dropna().unique()
        group_metrics = {}

        for group in groups:
            g = df[df[col] == group]
            n = len(g)
            n_flagged = g['predicted'].sum()
            n_actual_dropout = g[outcome_col].sum()

            # True positive rate (recall): among actual dropouts, what fraction flagged?
            tp = (g['predicted'] & g[outcome_col]).sum()
            fn = ((1 - g['predicted']) & g[outcome_col]).sum()
            tpr = tp / (tp + fn) if (tp + fn) > 0 else 0.0

            # Precision: among flagged students, what fraction actually dropped?
            fp = (g['predicted'] & (1 - g[outcome_col])).sum()
            precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0

            group_metrics[str(group)] = {
                'n': int(n),
                'flagging_rate': float(n_flagged / n) if n > 0 else 0.0,
                'actual_dropout_rate': float(n_actual_dropout / n) if n > 0 else 0.0,
                'true_positive_rate': float(tpr),
                'precision': float(precision)
            }

        if not group_metrics:
            continue

        flagging_rates = [m['flagging_rate'] for m in group_metrics.values()]
        tprs = [m['true_positive_rate'] for m in group_metrics.values()]
        precisions = [m['precision'] for m in group_metrics.values()]

        disparate_impact = min(flagging_rates) / max(flagging_rates) if max(flagging_rates) > 0 else 1.0
        tpr_gap = max(tprs) - min(tprs)
        precision_gap = max(precisions) - min(precisions)

        # Chi-squared test for flagging rate independence
        contingency_data = pd.crosstab(df[col], df['predicted'])
        chi2, p_chi2, _, _ = stats.chi2_contingency(contingency_data)

        results[col] = {
            'group_metrics': group_metrics,
            'disparate_impact_ratio': float(disparate_impact),
            'disparate_impact_flag': bool(disparate_impact < 0.8),  # 4/5ths rule
            'equal_opportunity_tpr_gap': float(tpr_gap),
            'predictive_parity_precision_gap': float(precision_gap),
            'flagging_rate_chi2_pvalue': float(p_chi2),
            'significant_disparity': bool(p_chi2 < 0.05),
            'recommendation': (
                "Investigate for bias - disparate impact below 80% threshold"
                if disparate_impact < 0.8 else "Acceptable disparate impact"
            )
        }

    return results


def intervention_recommendation(
    student_id: str,
    risk_score: float,
    shap_explanation: Dict,
    intervention_capacity: Dict[str, int]
) -> Dict:
    """
    Route a flagged student to the most appropriate intervention type
    based on their risk factors and available capacity.
    """
    risk_factor = shap_explanation.get('top_risk_factor', '')

    # Map risk factors to intervention types
    intervention_map = {
        'days_since_last_activity': 'engagement_outreach',
        'weeks_with_zero_activity': 'engagement_outreach',
        'assignment_submit_count': 'academic_support',
        'quiz_attempt_count': 'academic_support',
        'activity_trend_slope': 'advisor_meeting',
        'discussion_post_count': 'peer_connection'
    }

    intervention_type = intervention_map.get(risk_factor, 'advisor_meeting')

    # Check capacity
    if intervention_capacity.get(intervention_type, 0) <= 0:
        # Fall back to lower-resource intervention
        intervention_type = 'automated_email'

    return {
        'student_id': student_id,
        'risk_score': risk_score,
        'intervention_type': intervention_type,
        'priority': 'high' if risk_score > 0.75 else 'medium',
        'reason': shap_explanation.get('risk_factor_summary', [])
    }

---

Production Engineering Notes

The feedback loop is the hardest engineering problem. Prediction is the easy part. Closing the loop from prediction to intervention to outcome measurement requires: intervention protocols that advisors can execute consistently, outcome tracking that attributes improvement to the intervention, and a way to distinguish students who would have recovered without intervention from those who needed it (ideally via randomization).

Set thresholds based on operational constraints, not AUC. An advisor who can handle 25 flagged students per week needs a threshold that produces roughly 25 flags per week from your model, with a precision high enough to justify their attention. AUC does not tell you where to set the threshold. Use precision-recall curves and simulate the operational load at various thresholds before deployment.

Monitor prediction drift weekly. Student behavior changes with the academic calendar - activity spikes before exams and drops during breaks. A threshold set in week 3 may flag the wrong students in week 12. Recalibrate thresholds periodically and monitor whether the model's predictions remain calibrated as the semester progresses.

Do not use the same features for prediction and intervention evaluation. If your model uses "assignment submission rate" to predict dropout, and your intervention is reminders to submit assignments, then students who respond to the intervention will mechanically improve on the feature you used for prediction. You need holdout features or external assessments to evaluate whether interventions actually improved outcomes.

---

Common Mistakes

:::danger Deploying Predictions Without Intervention Infrastructure A dropout prediction model that flags students but has no systematic intervention protocol attached to it is worse than no model at all. It generates lists that overwhelm advising staff, creates false confidence that "something is being done," and may stigmatize students who are flagged but never contacted. Before deploying any student performance prediction system, define the intervention protocol, ensure advisor capacity is sufficient, and measure intervention outcomes. :::

:::danger Ignoring Demographic Bias in At-Risk Flagging Models trained on historical enrollment data inherit historical patterns, including structural disparities. A model trained on data where first-generation students historically had lower completion rates will flag first-generation students at higher rates. This can lead to differential surveillance without differential support. Always audit flagging rates across demographic groups and require that the intervention rate (actually received support) matches the flagging rate. :::

:::warning Using Only AUC to Select Threshold AUC measures model discrimination across all thresholds. The operational threshold - what risk score triggers a flag - should be selected based on advisor capacity, acceptable false positive rate, and the cost of missing a true at-risk student. A model with AUC 0.82 may be operationally inferior to one with AUC 0.79 if the 0.82 model requires 200 flags per week to achieve the same recall as the 0.79 model at 50 flags per week. :::

:::warning Confusing Prediction Timing with Intervention Timing A model that can predict dropout at week 2 of a 16-week course is more valuable than one that predicts accurately at week 12 - but only if the intervention at week 2 is effective. If the intervention requires 4 weeks to implement, a week 12 prediction gives only 4 weeks of intervention time. Match your prediction window to the time needed for an effective intervention to take effect. :::

---

Interview Questions and Answers

Q1: What features would you engineer from LMS clickstream data for dropout prediction, and which are most predictive?

From clickstream data I would engineer features in three categories: frequency (events per week, days active, assignment submission rate), recency (days since last login, days since last assignment submission), and trend (slope of weekly activity over time, ratio of last week activity to first week activity).

In practice, the most predictive features tend to be recency features - particularly days since last LMS activity and whether the student submitted the most recent assignment. These capture disengagement early. Trend features (is engagement declining?) are highly predictive after week 3. Absolute frequency features (total logins) are less predictive than trend and recency because they confound student behavior with course structure.

Missing data handling is critical: a student with no events for 7 days did not generate any clickstream data, but that absence is itself the strongest signal. Impute with zeros and create explicit "inactive period" features.

Q2: How would you design a fairness audit for a student dropout prediction system?

I would audit three fairness criteria: demographic parity (are flagging rates equal across groups?), equal opportunity (are true positive rates - the fraction of actual dropouts caught - equal across groups?), and predictive parity (when a student is flagged, does the probability they actually drop out equal the base rate, regardless of group?).

Practically: compute all three metrics across protected attributes (race/ethnicity, gender, first-generation status, Pell grant eligibility, disability status). Apply the 4/5ths rule: if any group's flagging rate is below 80% of the highest-flagging group's rate, investigate. For equal opportunity, compute the recall gap - the difference in true positive rates between groups. A large recall gap means the model is systematically missing at-risk students from one group.

The hard part: these criteria are mathematically incompatible (Chouldechova, 2017). If base rates differ across groups and you achieve predictive parity, you will necessarily violate demographic parity. You have to decide which criterion aligns with your intervention design, which requires policy input, not just technical input.

Q3: A student performance prediction model has high AUC but the intervention has no measurable effect on outcomes. What happened and what do you do?

Several possible causes. First, the intervention was not actually delivered to flagged students (implementation failure) - check whether flagged students were actually contacted and whether contact rates differ from the non-flagged group. Second, the intervention was delivered but is inherently ineffective - even the right students were contacted but the contact (a generic email saying "check in with your advisor") did not change behavior. Third, the prediction identified disengaged students but not students who can be helped by available interventions - some dropout is due to financial or family crises that advising cannot address.

The fix: run a randomized controlled trial. Randomly assign flagged students to intervention vs control. Measure outcomes in both groups. If the treatment group and control group have similar outcomes, the intervention is the problem, not the model. If the treatment group improves, the model was fine but the previous implementation lacked randomization and might have been selecting on characteristics correlated with recovery regardless of intervention.

Q4: What is survival analysis and when is it more useful than binary dropout prediction?

Binary dropout prediction answers "will this student drop out by end of semester?" Survival analysis answers "what is the probability this student drops out in any given week?" The survival function $S(t) = P(\text{enrolled at week } t)$ and hazard function $h(t)$ give a time-resolved view of dropout risk.

Survival analysis is more useful when: (1) intervention timing matters - if you want to contact students at the optimal moment before dropout becomes likely, you need week-level probability estimates, not just an end-of-semester flag; (2) your dataset has varying observation lengths - some students are observed for 4 weeks and some for 12 before the analysis date; binary prediction requires a fixed time horizon. Cox regression handles right-censored observations (students who have not yet dropped out) without discarding them.

The Cox model's hazard ratio interpretation is also useful for communication: "each additional day without LMS activity multiplies the dropout hazard by 1.08" (i.e., $\exp(0.077) = 1.08$) is a tangible, actionable statement for advisors.

Q5: How do you handle FERPA compliance for a student performance prediction system?

FERPA defines "education records" as records directly related to a student and maintained by the institution. Model-derived risk scores that are maintained in a student information system are likely FERPA records. Key compliance steps:

Data access control: risk scores visible only to school officials with a legitimate educational interest - typically the student's assigned advisor and relevant program staff. Not IT administrators, not research teams without IRB approval, not third-party vendors unless under FERPA agreements.

Purpose limitation: data collected for advising purposes cannot be repurposed for research, marketing, or other purposes without appropriate consent or authorization.

Student rights: students have the right to access their education records. If a student requests to see their risk score and the factors driving it, you must be able to provide this. This is a strong argument for explainable models - a SHAP explanation satisfies the disclosure requirement better than "the model said so."

Retention policy: when a student graduates or transfers, determine how long risk scores are retained. Best practice is to delete predictive model outputs when the administrative purpose is fulfilled.

Q6: How would you evaluate whether your early warning intervention system actually improved outcomes?

Gold standard: randomized controlled trial. Identify all flagged students, randomly assign half to receive the intervention (outreach, appointment, resources) and half to receive no outreach. Measure semester completion rates, final grades, and course registration for next semester in both groups. Any difference is attributable to the intervention.

Without randomization: interrupted time series analysis. If you deployed the system in fall 2023, compare dropout rates in your institution in fall 2023 vs fall 2022 (pre-system) while controlling for enrollment trends, economic conditions, and other time-varying factors. This is weaker evidence but feasible without randomization.

Also measure: the intervention delivery rate (what fraction of flagged students actually received outreach), the response rate (what fraction of contacted students took the recommended action), and the dose-response relationship (did students who completed more intervention components have better outcomes). These process measures help diagnose whether a null result is due to ineffective prediction, ineffective intervention, or implementation failure.

---

Summary

Student performance prediction turns LMS engagement data into actionable early warnings. The ML components are technically straightforward: feature engineering from clickstream events, LightGBM for dropout prediction, Cox regression for survival analysis, SHAP for explanations. The hard challenges are operational. Predictions without intervention capacity do not improve outcomes. Predictions without fairness audits may systematically under-serve already-disadvantaged students. FERPA compliance imposes real constraints on data handling. The measure of success is not model AUC - it is whether flagged students receive effective support and whether their outcomes improve compared to a control group. That requires closing the loop from prediction to intervention to outcome measurement, which is a product and process design challenge, not just an ML challenge.