ANOVA and Experimental Design - Comparing Multiple Models and A/B Tests
Master Analysis of Variance, F-statistics, one-way and two-way ANOVA, and rigorous A/B test design for ML model comparison and hyperparameter ablations.
Master Analysis of Variance, F-statistics, one-way and two-way ANOVA, and rigorous A/B test design for ML model comparison and hyperparameter ablations.
ACF and PACF functions, lag plots, correlograms, Ljung-Box test, and identifying ARIMA orders from autocorrelation structure. Essential for time series model selection in ML and forecasting.
Master bootstrap resampling, permutation tests, jackknife, and cross-validation as statistical tools for ML model evaluation. Build everything from scratch in NumPy.
Understand the potential outcomes framework, confounders, average treatment effect, difference-in-differences, and why offline evaluation of recommendation systems fails.
Master confidence intervals for ML engineering - correct interpretation of CIs, construction for means and proportions, bootstrap CIs, and uncertainty quantification for model evaluation metrics.
Master Maximum Likelihood Estimation and Maximum A Posteriori estimation. Understand why cross-entropy loss IS negative log-likelihood, and how bias-variance tradeoff applies to estimators.
Master hypothesis testing for ML engineering - correct interpretation of p-values, Type I/II errors, t-tests, chi-squared tests, and multiple testing corrections for model comparison.
Deep dive into linear regression OLS derivation, multiple regression, R-squared, logistic regression as a GLM, and Ridge/Lasso from a statistical perspective.
SciPy for machine learning - optimisation, sparse matrices, statistical distributions, signal processing, and distance metrics.
Master statistical power, effect size, sample size calculation, and power analysis for ML experiments and A/B tests. Know exactly when to stop an experiment and how many examples you need to detect model improvements.
How statistical theory powers ML model evaluation, A/B testing, and production AI systems. Module map, prerequisites, and learning objectives.