The ML Skills Timeline — AI Letters #04

2017

Attention Is All You Need

Vaswani et al. replace RNNs with self-attention. The transformer is born.

RNNs commoditized LSTMs commoditized Attention emerged

Engineers who had invested in LSTM sequence modeling suddenly needed to relearn their mental models. The RNN skill didn't transfer cleanly — attention is architecturally different. This was the first major commoditization wave for ML engineers who specialized in sequential modeling.

2018

BERT

Devlin et al. introduce bidirectional pre-training. Transfer learning becomes the standard for NLP.

TF-IDF classifiers Word2Vec pipelines Fine-tuning skills Contextual embeddings

BERT made feature engineering for NLP largely obsolete. The engineers who adapted focused on fine-tuning strategies and understanding what the model's representations actually meant. Those who didn't found their custom NLP pipelines quietly deprecated by their own teams.

2020

GPT-3 (175B)

Brown et al. demonstrate few-shot learning at scale. Kaplan scaling laws published. The "scale beats craft" thesis confirmed empirically.

Task-specific architectures Domain NLP models Prompt design In-context learning

The first time a single model could do dozens of NLP tasks without fine-tuning. Engineers who had built specialized models (question answering, summarization, translation) found GPT-3 competitive with their best systems — zero-shot. The correct response was to go deeper, not to compete at the task level.

2021

LoRA

Hu et al. show that large models can be adapted with <1% of parameters updated. Fine-tuning becomes affordable.

Full fine-tuning at scale Domain adaptation PEFT expertise

LoRA democratized fine-tuning. But it also raised the bar — now anyone could fine-tune, which meant the value shifted to engineers who could construct quality training data, design proper evaluation sets, and iterate on training runs systematically. The tooling got easier; the thinking got harder.

Nov 2022

ChatGPT

OpenAI releases ChatGPT. 100M users in 60 days. Every company starts an AI initiative overnight.

Custom NLP classifiers Rule-based NLP Basic BERT fine-tuning Systems thinking Eval design

The biggest single commoditization event in ML history. Text classification, NER, sentiment analysis, basic QA — all absorbed by GPT-3.5. The engineers who panicked were the ones whose entire value proposition was at the task layer. The engineers who accelerated were the ones who understood what ChatGPT couldn't do reliably yet.

Mar 2023

GPT-4 + RAG goes mainstream

GPT-4 ships. RAG becomes the dominant pattern for enterprise AI. Embedding + retrieval skills spike in demand.

Prompt-only pipelines Retrieval engineering Embedding evaluation RAG architecture

Everyone built a RAG system. Almost none of them worked reliably in production. The engineers who understood embedding spaces, retrieval failure modes, and context window mechanics became disproportionately valuable — because everyone else had a system that was failing and didn't know why.

2023–2024

Open weights + long context

Llama 2/3, Mistral, Gemini 1.5 (1M ctx). Fine-tuning open models becomes a production strategy, not a research project.

API-only strategies Fine-tuning + PEFT Attention mechanics Inference optimization

Open weights changed the competitive landscape. Teams that could fine-tune Llama-3 on proprietary data could beat GPT-4 on their specific domain — at 1/10th the cost. The skill premium shifted to engineers who could execute the full fine-tuning lifecycle: data curation, training stability, evaluation, deployment.

2025 →

Where we are now

Agentic systems, tool use, multimodal — all built on the same fundamentals. The seams are more complex, the stakes are higher.

Evals Retrieval Fine-tuning Attention mechanics Systems thinking

The fundamentals that survived every transition — understanding attention, evaluating models rigorously, reasoning about retrieval failures, fine-tuning for domains — are now the core of what it means to be an AI engineer. They didn't get commoditized. They compounded.