Module 0. Foundations before you start

Learn in the app: tutor, coding problems →

This is the entry point into the course "LLMs from scratch" for people who don't have the background yet. If you didn't pass the first diagnostic test and ended up here, that's fine. This is exactly what the module was written for. It takes 2 to 6 weeks and comes before the core of the course (that's what we call the main 24 weeks). How many weeks you need is decided not by how you rate yourself, but by the diagnostic test below: 60–90 minutes, pen and paper, no AI and no searching. The short test in the app is the same diagnostic, just in a shorter form (more on this below).

Workload: 15–20 hours a week. A Foundation week (that's the name of this module) is not built around lectures. It is short theory plus a lot of hands-on work: everything you learn here is taken for granted in the core from week 1 onward. There's no need to be afraid of that: every week below explains its terms from zero and gives an example with small numbers.

Who this is for

From week 1, the core of the course expects you to compute Jacobians (tables of derivatives of a function of many variables) and to do the backward pass by hand (computing gradients) through an MLP (a multi-layer network of linear layers and nonlinearities). If half of those words are unfamiliar right now, that's fine: this is precisely the case the module was written for. The core assumes four things already run on autopilot: Python without fighting the language, tensor shapes (the size of a multi-dimensional array of numbers along each axis) without guessing, derivatives of composite functions, and basic probability. On top of that, the core has no dedicated slot for classic ML, yet ML coding interviews sometimes ask you to implement logistic regression or k-means.

The module is for you if you:

come from software engineering and write code confidently, but last saw math at university;
come from science (physics, math, bioinformatics) and know the math, but write scripts rather than packages with tests;
learned ML on the surface through sklearn.fit() and have never implemented an algorithm yourself.

You do not need the module if you scored ≥80% on the diagnostic. Going through it "just in case" doesn't pay off: whatever gaps remain, the core will expose them on its own, and closing them one at a time is cheaper.

How the module works

Diagnostic (60–90 min)
   │
   ├─ ≥80%        → straight to week 1 of the core
   ├─ 50–80%      → only the F weeks that match your weak blocks
   └─ <50%        → all six weeks F1–F6
   │
Exit check → week 1 of the core

Week	Topic	Covers diagnostic block
F1	Python and engineering hygiene	1 (Python), 6 (systems thinking)
F2	NumPy and tensor shapes	2 (shapes and NumPy)
F3	PyTorch and autograd in practice	3 (derivatives), 6 (systems thinking)
F4	Linear algebra	2 (shapes and NumPy, questions 9–10)
F5	Calculus and probability	3 (derivatives), 4 (probability)
F6	Classic ML, the minimum	5 (basic ML)

Why this order: F1–F3 give you the tools, and F4–F6 give you the content. Linear algebra is easier to learn once NumPy is already in your hands and you can check every statement in two lines of code.

Diagnostic test

Rules. 30 questions, 6 blocks of 5. Answer in writing, briefly, with a derivation where one is needed. Score per question: 1 for a complete answer, 0.5 for a correct answer without explanation or an explanation with a small mistake, 0 for a wrong answer or "I don't know". Don't guess: "I don't know" is more honest and more useful, because it sends you to the right week.

The answer keys come after all the questions. Don't open them until you're done.

Two forms of the same test. In the app the diagnostic is shorter: 24 multiple-choice questions, 4 per block, about 10 minutes. Here is the full paper version: 30 open questions that you have to solve, not just recognize. The blocks, the thresholds (80% / 50%) and the recommended paths are the same for both forms. Take the short one for quick orientation, and take the full one if your result lands near a threshold.

Block 1. Python

What does this code print, and why?

def add(x, acc=[]):
    acc.append(x)
    return acc
add(1)
print(add(2))

What does sum(x * x for x in range(4)) return? How much memory does the same expression take with range(10**9)? Does the answer change if you replace the parentheses with square brackets?
Write a decorator timed that prints the running time of any function and returns its result unchanged.
What does this code print, and why?
```
a = [[0] * 3] * 3
a[0][0] = 1
print(a)
```
A script runs for 40 minutes. How do you find out which function is eating the time? Name the tool and the command.

Block 2. Tensor shapes, NumPy, linear algebra

A.shape == (32, 10), b.shape == (10,). What is the shape of A + b? What happens with A + b[:, None]?
X.shape == (B, D), W.shape == (D, F). What is the shape of X @ W, and how many multiply-add operations are performed?
x = np.arange(12).reshape(3, 4), y = x.T. Does .T copy the data? What is y[0, 0] after x[0, 0] = 100?
What is the rank of the matrix u vᵀ for nonzero vectors u ∈ ℝⁿ, v ∈ ℝᵐ?
The matrix Q is orthogonal. What are ‖Qx‖ and Q⁻¹?

Block 3. Derivatives

Find the derivative of σ(x) = 1 / (1 + e^{−x}) and express it in terms of σ(x).
Find the derivative of f(x) = log(1 + eˣ).
Find the gradient of f(x) = xᵀAx with respect to the vector x. How does it simplify for a symmetric A?
Find the gradient of L(w) = (wᵀx − y)² with respect to w.
L = sin(u), u = x². Find dL/dx.

Block 4. Probability

A fair coin is tossed 3 times. What is the probability of at least one heads?
What are the expected value and the variance of Binomial(n, p)?
A disease affects 1% of people. A test comes back positive for 99% of sick people and for 1% of healthy people. A person tests positive. What is the probability that they are sick?
X and Y are independent, Var(X) = Var(Y) = 1. What is Var(X − Y)?
A coin lands heads with probability p. On average, how many tosses do you need up to and including the first heads?

Block 5. Basic ML

Training loss goes down, validation loss goes up. What is happening, and what are three things you can do about it?
The positive class is 1% of the data. The model always predicts "negative". What is its accuracy? Why is that bad, and which metric should you look at instead?
Why do you need a separate test set if you already have a validation set?
Which loss function does logistic regression minimize, and why not MSE?
Give two concrete examples of data leakage.

Block 6. Systems thinking

A model with 1B parameters. How much memory do its weights take in fp32? In bf16?
Training has been running for 10 hours and GPU utilization stays at 20%. Where do you look for the cause first?
Name four things without which an experiment can't be reproduced a month from now.
What does git bisect do, and in what situation is it irreplaceable?
The training loop calls loss.item() on every step to print it. Why can this noticeably slow down training on a GPU?

Answer keys

Block 1.

[1, 2]. The default value is evaluated once, when the function is defined, and the list lives on between calls. The right way: acc=None and inside if acc is None: acc = [].
14 (0+1+4+9). A generator is lazy: O(1) memory for any range. With a list the answer is the same, but first a list of 10⁹ elements is materialized, and that is tens of gigabytes.
Key elements: functools.wraps(fn), a wrapper with *args, **kwargs, time.perf_counter() before and after, return result. Without wraps the function loses its name and docstring; without return the decorator breaks any function that returns something.
[[1, 0, 0], [1, 0, 0], [1, 0, 0]]. Multiplying a list copies references, not objects: all three rows point to the same list. The right way: [[0] * 3 for _ in range(3)].
python -m cProfile -s cumtime script.py (or line_profiler for line-by-line). Any profiler counts. "I'll sprinkle prints with timestamps" doesn't count: that measures what you already suspect instead of searching.

Block 2.

(32, 10): b is stretched along the first axis. b[:, None] has shape (10, 1); comparing from the right: 1 and 10 are compatible, 10 and 32 are not → broadcasting error.
(B, F); B·D·F multiply-adds, that is 2BDF FLOPs. This is the same factor of two that turns into 2N per token in week 9.
It doesn't copy: .T returns a view with permuted strides. y[0, 0] == 100.
Rank 1: every column equals u times a number.
‖Qx‖ = ‖x‖ (orthogonal transformations preserve lengths), Q⁻¹ = Qᵀ.

Block 3.

σ'(x) = σ(x)(1 − σ(x)).
f'(x) = eˣ / (1 + eˣ) = σ(x). Worth remembering: the derivative of softplus is the sigmoid.
∇f = (A + Aᵀ)x; for a symmetric A this becomes 2Ax.
∇L = 2(wᵀx − y)x.
dL/dx = 2x·cos(x²).

Block 4.

1 − (1/2)³ = 7/8.
E = np, Var = np(1 − p): it is a sum of n independent Bernoulli variables.
0.5. Sick and positive: 0.01 · 0.99 = 0.0099; healthy and positive: 0.99 · 0.01 = 0.0099. The answer "99%" is the classic mistake: the base rate was ignored.
2. Variances of independent variables add up even when you subtract: Var(−Y) = Var(Y).
1/p: the expected value of the geometric distribution.

Block 5.

Overfitting. Actions: regularization (weight decay, dropout), early stopping, more data or augmentation, a smaller model. Any three count.
99%. The model is useless: it doesn't find a single positive. Look at recall, precision, F1, PR-AUC; with strong imbalance, accuracy tells you nothing.
You use validation to pick hyperparameters and the model, so the estimate on it is optimistically biased. You touch the test set once, at the very end.
Binary cross-entropy, that is, the negative log-likelihood of a Bernoulli. With a sigmoid, MSE gives a non-convex problem and a near-zero gradient on a confident mistake (the sigmoid is saturated). Either argument counts if it's explained.
For example: normalizing over the whole dataset before the train/test split; duplicates or near-duplicates between train and test; a feature computed from information from the future; the same user or patient in both train and test.

Block 6.

fp32: 4 bytes × 10⁹ = 4 GB; bf16: 2 GB.
The dataloader (the CPU can't prepare batches fast enough), CPU–GPU synchronizations, a batch that's too small. And most importantly: don't guess, run a profiler.
Fixed seeds; code versions (commit hash) and dependency versions; a config with all hyperparameters; the data version. Any four reasonable answers count.
Binary search over the commit history: it finds the commit that introduced a regression in log₂(n) checks. Irreplaceable when "it used to work" and there are hundreds of commits between "worked" and "broke".
.item() copies the value to the CPU and forces the CPU to wait until the GPU finishes all queued kernels. Asynchrony is lost: the GPU sits idle while the CPU prepares the next step.

How to score yourself

Total (out of 30)	Share	What to do
≥ 24	≥ 80%	Straight to week 1 of the core
15–23.5	50–80%	Selectively: every block where you scored ≤ 3 out of 5 unlocks its F weeks from the table above
< 15	< 50%	The whole module, F1–F6 in order

Borderline case: total score ≥ 24, but one block ≤ 2 out of 5. Do only the matching F week: a single failure points to a specific gap, not to your overall level.

Week F1. Python and engineering hygiene

The goal of the week: the language and the tools stop stealing your attention. If you already write Python, this week will go fast. If you don't, nothing here is especially hard either: you need only a few things, and each one can be checked in a minute in the console. In the core every assignment is set up as a package (a folder of code that Python knows how to install and import) with tests (small programs that check that the code gives the expected answer). If pip install -e . (the command that installs a package in development mode) or a failing import eats half an hour, you never get to the actual content.

Theory

In plain terms. a = [1, 2], then b = a, then b.append(3). Now a is [1, 2, 3] too: there is one list, and it has two names. Assignment doesn't copy. It attaches one more name to the object. The trap in question 1 works the same way: acc=[] creates one list at the moment def runs, and every call without that argument appends to that same list.

The data model. A variable is just a name bound to an object; assignment doesn't copy. Mutable (list, dict, tensors) and immutable (int, str, tuple) objects. From this you can see that both diagnostic traps (questions 1 and 4) are really the same trap

In plain terms. A 50 GB file, a 1 MB window. A list of all windows would try to hold all 50,000 windows in memory at once. A generator with yield hands out one window, waits until it's processed, and only then reads the next one. There is always one window in memory, 1 MB, for a file of any size.

Iterators and generators. yield turns a function into a lazy (computed on demand) stream. The takeaway: a dataloader (the code that feeds data into the model) that reads a 50 GB corpus in batches (chunks of examples) must be a generator, not a list

In plain terms. Writing @timed above def f(...) means exactly f = timed(f) right after the definition: the name f now points to the wrapper, and the original function lives inside it. A with obj: block calls obj.__enter__() on entry and obj.__exit__(...) on exit, even if an exception happened inside.

Decorators and context managers. @torch.no_grad() and with torch.no_grad(): use the same object in two ways. Once you understand how that's possible, you understand both mechanisms
Classes and dataclasses. A model config as a @dataclass gives you types, default values and a repr for free
Packages. pyproject.toml, a virtual environment (a separate set of libraries for one project), pip install -e .. Why an editable install: tests import the package the same way a user does, and code edits show up without reinstalling
Tests. pytest, assert, fixtures (prepared data and objects that pytest passes into a test), -k to select tests. A test works as an executable specification: all of nanolm is built so that an assignment counts as done when the reference tests pass
Profile before you optimize. cProfile and line_profiler. Your intuition about where code is slow is wrong more often than you think
Git. commit, branch, merge vs rebase, resolving a conflict, .gitignore for weights and data (weights in git: the most common hygiene mistake in ML repositories)
Shell and remote work. pipes, grep, find, environment variables, ssh keys, tmux: the session survives a dropped connection, and training doesn't die along with your laptop

Practice. Create a foundation/ package with a pyproject.toml; in it, a generator that reads a large text file in fixed-length windows without loading it whole; a timed decorator; three functions with tests. Find a deliberately slow function with cProfile, speed it up and show the difference as a number. Everything under git, with meaningful commits and one branch merged in via rebase.

Week outcomes

I can create a package with pyproject.toml from scratch in 10 minutes, install it in editable mode and run pytest.
I can write a generator that reads a file of any size in windows and explain why memory stays O(1).
I can use cProfile to find the most expensive function in an unfamiliar script and confirm the speedup with a measurement.
I can create a branch, rebase it onto a fresh main and resolve a conflict without losing changes.

Self-check

Why is a mutable default value a trap, and how do you avoid it?
How can torch.no_grad() work both as a decorator and with with?
How does an editable install differ from a regular one, and why do you want it in a project with tests?

Resources: The Python Tutorial (docs.python.org); Luciano Ramalho, Fluent Python (2nd ed., 2022), the chapters on the data model, iterators and decorators; Pro Git (Scott Chacon, Ben Straub); MIT The Missing Semester of Your CS Education.

Week F2. NumPy and tensor shapes

NumPy (the library for working with arrays of numbers) and tensors (multi-dimensional arrays: a vector, a table, a stack of tables) sit underneath all the code in the course. The main skill of the week: predicting the shape of a result (the size along each axis, for example (32, 10)) before you run the code. Half of all bugs in ML code come from broadcasting that silently kicked in (the automatic stretching of arrays to a common shape) and produced a tensor of the right length and the wrong meaning. It sounds scary, but there are only two rules, and they're worked through with numbers below.

Theory

In plain terms. A 3×4 array of int64 sits in memory as a single strip of 12 numbers, 8 bytes each. To move to the next column you shift by 8 bytes, to the next row by 32: strides = (32, 8). .T doesn't touch the strip. It only swaps the steps: strides = (8, 32). Same buffer, different rule. That's why transposing is free.

What an array is. A memory buffer + shape + strides + dtype. strides say how many bytes to shift to get to the next element along an axis
View (a new look at the same buffer, without copying) vs copy. Transposing, slicing and reshape of a contiguous array give a view: only shape and strides change. A takeaway you'll need in week 1 of the core: transposing is free, so you can store W in either orientation
Contiguity. After .T the array isn't contiguous; some operations need a copy (np.ascontiguousarray, in PyTorch .contiguous(), and that's why .view sometimes fails while .reshape doesn't)

In plain terms. a of shape (3, 1), the column [0, 10, 20]; b of shape (1, 4), the row [1, 2, 3, 4]. Shapes are aligned from the right: 1 and 4 → 4, 3 and 1 → 3. The result has shape (3, 4), a table of every-with-every sums: [[1, 2, 3, 4], [11, 12, 13, 14], [21, 22, 23, 24]]. The same mechanics produce the trap. x = [1, 2, 3] of shape (3,) minus the same thing of shape (3, 1) was supposed to give three zeros, but gives a 3×3 table of all pairwise differences.

Broadcasting (automatic stretching of size-1 axes). Shapes are aligned from the right, a size-1 axis is stretched, anything else is an error. The main trap: (B,) minus (B, 1) gives (B, B) without a single warning. The antidote: assert x.shape == (...) after every non-trivial operation

In plain terms. x of shape (2, 3), x.max(axis=1) has shape (2,): one maximum per row. x − m aligns (2, 3) and (2,) from the right: 3 vs 2, error. It's worse if x is square, (3, 3): no error, but element [i, j] gets the maximum of row j subtracted, not of row i. With keepdims=True, m has shape (2, 1), and each row subtracts its own maximum.

Reductions along an axis and keepdims. x - x.max(axis=1, keepdims=True): the first step of a stable softmax (the function that turns a set of numbers into probabilities); without keepdims rows get subtracted from the wrong rows
Indexing. Basic indexing (slices) returns a view, "fancy" indexing (index arrays, masks) returns a copy. x[np.arange(B), targets] picks the logit (the raw output of the model before softmax) of the correct class in each row, the basis of cross-entropy
Vectorization. A Python loop spends interpreter work on every element, a vector operation makes one call into a loop in C. A 50–200× difference isn't an optimization. It's the condition for your experiments to finish at all
einsum as a language of shapes. 'bij,bjk->bik' writes a batched matmul; 'bhqd,bhkd->bhqk' writes attention scores. If you can write an operation in einsum, you know its shapes for sure

Practice. Without a single loop over the data: a pairwise distance matrix via ‖a‖² + ‖b‖² − 2aᵀb; one-hot via indexing; a stable softmax along an arbitrary axis; a batched matmul via einsum, checked against @. For the pairwise distances, take a measurement: loop vs vectorization at 2000×2000, write down the time ratio. Every line of code gets a comment with the shape of its result.

Week outcomes

I can predict the result shape of any chain of broadcasting operations and say in advance where the error will be.
I can implement a pairwise distance matrix without loops and explain where the −2aᵀb comes from.
I can explain via strides why .T doesn't copy data while fancy indexing does.
I can rewrite any operation from attention in einsum and back.

Self-check

What does (B,) − (B, 1) produce, and how do you catch this kind of bug automatically?
Why keepdims=True in a stable softmax?
Why is vectorized code tens of times faster than a loop if it computes the same thing?

Resources: the NumPy documentation, the sections NumPy: the absolute basics for beginners and Broadcasting; the numpy.einsum documentation.

Week F3. PyTorch and autograd in practice

PyTorch (the main library for training neural networks) can compute the derivatives of any program by itself. This mechanism is called autograd (automatic differentiation): you write the forward pass, that is, the computation of the model's answer, and it builds the gradients for you (the derivatives of the loss, a number that measures the error, with respect to each parameter). Week 2 of the core will ask you to write your own autograd. Here the task is the reverse: learn to use the ready-made one in a way that lets you understand what it does and catch it when something goes wrong.

In plain terms. x = torch.tensor(2.0, requires_grad=True), y = x * x, y.backward(). Now x.grad equals 4: PyTorch remembered that y was obtained by squaring x, and applied the rule (x²)' = 2x at the point 2. No formulas by hand, just the chain of operations it recorded along the way.

Theory

Tensor = NumPy array + device + graph. device, dtype, moving between CPU and GPU, and what that move costs
Dynamic graph. The graph is built during the forward pass, anew with every operation. That's why you can write ordinary if statements and loops inside a model

In plain terms. w = 1.0, loss L = 3·w. After the first backward(), w.grad = 3. After a second one without zero_grad(), w.grad = 6: the new gradient was added to the old one. If this is an accident, on step two the optimizer sees the sum of two gradients, on step three of three, and the steps keep growing. If it's on purpose, it's accumulation: 4 microbatches of 8 examples, backward() after each (loss divided by 4), one step(). That's the same as one batch of 32.

.backward() accumulates gradients in .grad rather than overwriting them. Hence the mandatory zero_grad(), and hence also free gradient accumulation over several microbatches, which you'll need in week 10
no_grad, inference_mode, detach. Three ways to say "no graph needed here", with different levels of strictness
nn.Module. parameters(), state_dict, registering submodules, train() and eval(), and why a forgotten eval() changes the result (dropout, batchnorm)
Dataset and DataLoader. num_workers, pin_memory, shuffle: where data is prepared in parallel with training
The canonical training loop. Five lines you write on autopilot: forward → loss → zero_grad → backward → step

In plain terms. A batch of 8 examples and a model with thousands of parameters: it has to memorize those 8, and the loss drops almost to zero within a hundred steps. If the loss on an 8-class problem is stuck around ln 8 ≈ 2.08, the model is predicting all classes equally, which means it isn't learning at all. Look for a bug in the code (a forgotten step, mixed-up labels, a detached graph), don't tune the learning rate.

Debugging. The first test for any model: overfit a single batch to a loss near zero. If you can't, the bug is in the code, not in the hyperparameters. The second test: check the gradient with torch.autograd.gradcheck in float64
Synchronization. CUDA is asynchronous (the CPU queues kernels and moves on without waiting for the GPU); .item(), .cpu(), print(tensor) force the CPU to wait for the GPU (question 30 of the diagnostic)

Practice. Logistic regression two ways: a gradient written out by hand and a gradient from autograd, matching to 1e-6. Then an MLP on MNIST (or any small dataset) with a Dataset, a DataLoader, a training loop, validation, and saving and loading a state_dict. Separately, show that the model can overfit a single batch, and that a forgotten zero_grad() breaks training (write down exactly how).

Week outcomes

I can write a training loop with validation and a checkpoint from an empty file in 15 minutes.
I can check a hand-written gradient against autograd and explain the discrepancy, if there is one.
I can explain why .backward() accumulates gradients and use that for accumulation over microbatches.
I can diagnose a broken model with the single-batch overfitting test.

Self-check

What happens if you forget optimizer.zero_grad()? Describe how the loss behaves.
How do detach(), torch.no_grad() and torch.inference_mode() differ?
Why is gradient checking done in float64?

Resources: the official PyTorch Tutorials, the Learn the Basics series and A Gentle Introduction to torch.autograd; Andrej Karpathy, the video The spelled-out intro to neural networks and backpropagation: building micrograd.

Week F4. Linear algebra

Linear algebra sounds scarier than it is. This isn't a full course, just exactly what you'll need in the core: matrix multiplication from four angles, rank (how many independent directions a matrix has), orthogonality (when a transformation preserves lengths and angles), the spectral decomposition and SVD (two ways to break a matrix into simple factors; each one is shown with numbers below). Every statement this week can be checked in NumPy in two lines. That's how you should learn it: didn't believe a formula, wrote two lines, saw the numbers.

Theory

A matrix defines a linear map, and a product defines a composition. Hence non-commutativity, and why W₁W₂x = Wx (week 1 of the core)

In plain terms. A = [[1, 2], [3, 4]], B = [[5, 6], [7, 8]], AB = [[19, 22], [43, 50]]. Row times column: 19 = 1·5 + 2·7. Sum of outer products: the first column of A times the first row of B gives [[5, 6], [15, 18]], the second column times the second row gives [[14, 16], [28, 32]]; the sum is again [[19, 22], [43, 50]]. In ∂L/∂W = XᵀG the columns of Xᵀ correspond to the examples in the batch, and each term is the contribution of one example.

Four views of AB: dot products of rows with columns; the columns of AB as combinations of the columns of A; the rows as combinations of the rows of B; **a sum of outer products** of the columns of A with the rows of B. The last view explains why the weight gradient Xᵀ G is a sum of contributions from the individual examples in the batch
Rank, image, kernel. Rank equals the number of independent directions. uvᵀ has rank 1, BA with B ∈ ℝ^{d×r} has rank at most r. That's the whole idea of LoRA (week 15)

In plain terms. A 90° rotation: Q = [[0, −1], [1, 0]]. Q·(3, 4) = (−4, 3): the length was 5 and stayed 5. The reverse rotation is Qᵀ = [[0, 1], [−1, 0]], and that's also Q⁻¹: inverting an orthogonal matrix is free.

Orthogonality and projections. Projection onto a subspace; orthogonal matrices preserve lengths and angles. The rotation in RoPE (week 7) is given by an orthogonal matrix, so it doesn't change the norm of q and k
Eigenvectors of symmetric matrices. The spectral theorem: A = QΛQᵀ with an orthogonal Q. Positive semidefiniteness
SVD: A = UΣVᵀ for any matrix. The best rank-k approximation in the Frobenius norm comes from keeping the k largest singular values; the error equals the square root of the sum of squares of the discarded ones. PCA (F6) and low-rank methods are built on this

In plain terms. f(x) = ½(x₁² + 100·x₂²), Hessian diag(1, 100), condition number κ = 100 / 1 = 100. Gradient descent multiplies each coordinate by (1 − η·λ): x₁ by 1 − η, x₂ by 1 − 100η. With η = 0.019 the factor for x₂ is −0.9, so it converges. But x₁ shrinks by a factor of only 0.981 per step, and getting to 10⁻⁶ takes about 720 steps. With η = 0.021 the factor for x₂ is −1.1: x₂ grows by 10% per step and in 10 steps increases 2.6 times. The steep direction sets the step-size limit, and the flat direction sets the speed.

Norms and conditioning. The Frobenius and spectral norms; the condition number (κ = λ_max / λ_min for a symmetric positive definite matrix) shows how much a matrix stretches errors. The link to training: for a quadratic function, gradient descent converges only when η < 2/λ_max, where λ_max is the largest eigenvalue of the Hessian

Practice. Verify the four views of matrix multiplication numerically. Compress any matrix (for example, a grayscale image) with SVD for k = 1…50, plot the error against k, and compare it with the sum of the discarded σ². Implement the power method for the top eigenvector and compare it with np.linalg.eigh. Run gradient descent on a quadratic function with η slightly below and slightly above 2/λ_max and watch it converge and diverge.

Extra: conditioning with numbers. For a quadratic objective, gradient descent with the optimal step η = 2/(λ_min + λ_max) converges at a rate of (κ − 1)/(κ + 1) per step. With κ = 100, reaching 10⁻⁶ accuracy takes on the order of 700 steps, with κ = 10 about 70. Two practical tricks follow: feature normalization works as preconditioning (it squeezes the spectrum), and ridge regularization shifts all eigenvalues by λ and lowers κ. For practice: compare κ = 1 and κ = 10 and check the number of steps against the prediction. This is the trainer problem gd_steps_to_tol.

Gram–Schmidt, QR, and why least squares isn't solved via (AᵀA)⁻¹. One line is enough: κ(AᵀA) = κ(A)², so the normal equations double the loss of precision, while QR or SVD work with A itself. Optional: least squares three ways on an ill-conditioned matrix, and a comparison of the errors.

Week outcomes

I can explain matrix multiplication in all four ways and show which one gives ∂L/∂W = XᵀG.
I can implement a low-rank approximation via SVD and predict its error before running it.
I can implement the power method and explain what its convergence speed depends on.
I can derive the bound η < 2/λ_max for a quadratic function and confirm it with an experiment.

Self-check

Why is the rank of BA at most r, and what does that mean for LoRA?
How are the SVD of a data matrix and the eigenvectors of its covariance matrix related?
Why does a learning rate that's too large lead to divergence? Explain it through eigenvalues.

Resources: Li, Mathematical Pathways to Machine Learning (2026), ch. 11, pp. 71–75, exercises 11.1–11.6 (conditioning and convergence speed); 4.2–4.2.1 and 9.3.2 (QR and least squares) (CC BY-NC license: link and problem numbers only); Gilbert Strang, Introduction to Linear Algebra; MIT course 18.06 (Strang); 3Blue1Brown, Essence of linear algebra; Deisenroth, Faisal, Ong, Mathematics for Machine Learning (2020), chapters 2–4.

Week F5. Calculus and probability

Two halves of one week. Derivatives (the rate of change of a function) are needed for training: a model learns by moving its parameters in the direction where the loss decreases fastest. Probability is needed for the loss and for evaluation: a model doesn't predict "the correct token", it predicts a probability distribution over all tokens, and you need to be able to measure the quality of such a prediction. If school calculus has faded, that's no problem: everything you need is derived again below, with small numbers. Track D in the core then goes deeper into probability through problems; this week gives you the language you need to even read those problems.

Calculus

In plain terms. f(x) = x² at the point 3. Step by 0.01: (3.01² − 9) / 0.01 = 6.01. By 0.001: 6.001. The slope tends to 6 = 2·3. The extra part equals the step itself: ((x + h)² − x²)/h = 2x + h. The central difference (3.01² − 2.99²)/0.02 gives exactly 6: the first-order errors cancel. Hence f(3.01) ≈ 9 + 6·0.01 = 9.06, and exactly it's 9.0601: the derivative is the best linear approximation.

The derivative as the best linear approximation. f(x + h) ≈ f(x) + f'(x)h. In many dimensions the gradient or the Jacobian plays the role of f'
Why the gradient points in the direction of steepest ascent. The change ∇fᵀh with ‖h‖ = 1 is largest when h points the same way as ∇f (the Cauchy–Schwarz inequality). It's one line, and it holds the whole motivation for gradient descent
The chain rule for scalars, then for vectors: a product of Jacobians
Second-order Taylor expansion f(x+h) ≈ f + ∇fᵀh + ½hᵀHh: where the step-size limit comes from, and why curvature (the Hessian) matters
Convexity. For a convex function, every local minimum is global. Logistic regression is convex, a neural network is not
Exponent and logarithm. log(ab) = log a + log b, the derivative of log, why loss is computed in logs (a product of probabilities over tokens → a sum)

In plain terms. You split 9 hours a week between reading, x, and code, y, and the benefit is 2 ln x + ln y. At the optimum the last hour brings the same benefit in both activities: 2/x = 1/y, that is, x = 2y. Together with x + y = 9 you get x = 6, y = 3. That shared benefit of the last hour, λ = 2/6 = 1/3, is the Lagrange multiplier. Give yourself 10 hours instead of 9: the optimum is x = 20/3, y = 10/3, and the benefit grew by 0.316, almost exactly λ ≈ 0.333.

Constrained optimization. Maximize f subject to g = c: at the optimum ∇f = λ∇g, the gradient of the objective is parallel to the gradient of the constraint (otherwise you could move along the constraint and gain). The multiplier λ works as a shadow price (what one extra unit of the resource is worth): it's how much the best result grows if you relax the constraint by one unit. You'll meet it in Track D: D11 (probabilities sum to 1) and D20 (compute budget)

Probability

Random variable, PMF (the probability of each value of a discrete variable), PDF (the density of a continuous one), CDF (P(X ≤ x)). Expected value and variance
Linearity of expectation works without independence. The single most useful fact in counting problems

In plain terms. 10,000 people, 1% are sick, that is 100. The test catches 99 of them. Of the 9,900 healthy people, 1% test falsely positive, that is 99 people. There are 198 positives in total, and 99 of them are sick: half, not 99%. If 10% were sick, then out of 1,080 positives 990 would be sick, that is 92%. The test's accuracy is the same. What changed is the base rate (the share of sick people before testing).

Conditional probability, Bayes' rule and the base rate (question 18 of the diagnostic)
Distributions: Bernoulli, binomial, categorical, normal. The categorical distribution is exactly the output of softmax
LLN and CLT at the level of consequences. The standard error of the mean is σ/√n. The takeaway: accuracy on 100 examples has noise on the order of ±5 percentage points, and a 2-point difference between models on such a set doesn't count as a result (week 18 of the core)

In plain terms. 10 tosses, 7 heads. The probability of exactly this sequence when heads has probability p is p⁷(1 − p)³. At p = 0.5 it's 0.00098, at p = 0.7 already 0.00222, at p = 0.8 smaller again, 0.00168. The maximum is at p = 0.7 = 7/10. That's MLE (maximum likelihood estimation): pick the parameter under which what you observed is most probable.

MLE. For a coin: maximize pᵏ(1−p)^{n−k} → p̂ = k/n. Minimizing cross-entropy when training an LM is MLE written from the other side

In plain terms. The phrase "I drink tea". Say a text starts with the word "I" with probability 0.1. After "I" the word "drink" comes with probability 0.2, and after "I drink" the word "tea" comes with probability 0.3. The probability of the whole phrase: 0.1 · 0.2 · 0.3 = 0.006.

The chain rule of probability. p(a, b, c) = p(a) · p(b | a) · p(c | a, b): a joint probability (that everything happened together) can be built from conditional ones, adding one element at a time. It's an identity: it holds for any distribution, with no assumptions. A language model works exactly like this: at every step it outputs p(next word | everything before it), and the probability of the text is the product. To avoid drowning in tiny numbers, you add logarithms: ln 0.1 + ln 0.2 + ln 0.3 ≈ −5.12, exactly ln 0.006. The LM loss is this sum with a minus sign (week 4)

In plain terms. X is a Gaussian with mean 1 and variance 4 (standard deviation 2). Take Y = 3X + 2. The mean behaves like an ordinary number: 3·1 + 2 = 5. The spread stretches threefold: standard deviation 6, variance 9·4 = 36. Shifting by 2 doesn't change the spread. And Y is again a Gaussian: the bell just moved and got wider.

Affine transformation of a Gaussian (affine means "multiply and add"). If X ~ N(μ, σ²), then aX + b ~ N(aμ + b, a²σ²). The sum of two independent Gaussians is also a Gaussian, and the variances add. Hence the 1/√d scale in attention (week 7): a sum of d independent terms has d times the variance

In plain terms. A rubber band with dots drawn on it: stretch it to twice its length, and there are just as many dots, but half as many per centimeter. Density works the same way. X ~ N(0, 1), the density at zero is ≈ 0.399. For Y = 2X the same probabilities are spread over intervals twice as long, so the density at zero is 0.399 / 2 ≈ 0.199. Check against the rule above: Y ~ N(0, 4), its density at zero is 1/(2·√(2π)) ≈ 0.199. It matches.

Change of variables for a density. If Y = g(X) and g is monotonic, then p_Y(y) = p_X(x) · |dx/dy|, where x = g⁻¹(y). The factor |dx/dy| corrects for the stretching. In many dimensions its place is taken by the absolute value of the determinant of the Jacobian (week F4). Probabilities are preserved under a change of variables, densities are not

In plain terms. You have two coins in your pocket: a fair one (heads with probability 0.5) and a bent one (heads 0.9). You pull one out at random, 50/50, toss it and show only the result. Which coin is in your hand can't be seen: that's a latent (hidden) variable. The probability of heads is 0.5·0.5 + 0.5·0.9 = 0.7. It came up heads: by Bayes, the coin is the bent one with probability 0.45 / 0.7 ≈ 0.64.

Latent variable and marginalization. p(x) = Σ_z p(z) · p(x | z): to find the probability of what you can see, you sum over all values of the hidden z (marginalization, "summing out the hidden part"). If you take two Gaussians instead of two coins, you get a Gaussian mixture: first a coin picks a Gaussian, then a number is drawn from it. Many generative image models work this way: an image has a hidden code it is born from (week 20)

Practice. Check the derivatives from the diagnostic numerically with finite differences. Monte Carlo: simulate the disease-test problem and compare with Bayes' rule; simulate the mean of n tosses for n = 10, 100, 1000 and see 1/√n on a plot of the spread. Derive the MLE for the mean and variance of a normal distribution.

Critical points via the Hessian. The signs of the Hessian's eigenvalues at a point with zero gradient tell you what you're looking at: all positive means a minimum, all negative a maximum, mixed signs a saddle. In deep networks there are incomparably more saddles than local minima, and that's one of the reasons why "getting stuck in a local minimum" isn't the main problem in training.

Week outcomes

I can derive the gradients of xᵀAx, (wᵀx − y)² and the logistic loss without hints.
I can explain in 2 minutes why the negative gradient points in the direction of steepest descent.
I can solve a Bayes' rule problem and check the answer with a simulation.
I can estimate the noise of accuracy on a test set of size n and say whether a difference is significant.

Self-check

Why does linearity of expectation not require independence, while adding variances does?
Derive the MLE for the probability of heads for a coin.
What does the second-order Taylor expansion tell you about choosing the gradient descent step size?

Resources: Li, Mathematical Pathways to ML (2026), ch. 7 (especially 7.4 and example 7.7, backprop through a small network with numbers) and ch. 8, exercises 7.1–7.5: a lighter alternative to MML if the diagnostic gave < 50% (CC BY-NC: link only); Deisenroth, Faisal, Ong, Mathematics for Machine Learning (2020), chapters 5–6; 3Blue1Brown, Essence of calculus; Joseph Blitzstein, Jessica Hwang, Introduction to Probability (2nd ed., 2019) and the Harvard Stat 110 course.

Week F6. Classic ML, the minimum

Classic ML (machine learning before neural networks: simple models that fit entirely on one page) is boiled down here to exactly five things: logistic regression (predicting "yes/no" from a weighted sum of features), k-means (splitting points into k piles), PCA (finding the directions along which the data is most spread out), the decision tree (a chain of "is the feature above the threshold?" questions) and classification metrics (the numbers you use to judge whether a model is any good). Everything is implemented in NumPy without sklearn (the library with ready-made implementations); sklearn is allowed only for checking your answer. If you've never written a single algorithm yourself, this is the best week to start: each of them takes 20–40 lines.

Why only these: in ML coding interviews for LLM roles you're rarely asked to implement a classic algorithm, and when you are, it's almost always one of the four algorithms above. They're short, and they test your command of shapes and optimization. SVMs, boosting and naive Bayes are deliberately not implemented here: their cost in hours is out of proportion to the chance of being asked. For a general ML round it's enough to explain boosting out loud: a short review is at the end of the week.

Theory

Logistic regression. p = σ(wᵀx + b), loss: binary cross-entropy. The weight gradient Xᵀ(p − y)/n has the same "prediction minus target" form as softmax + CE in week 2 of the core. The problem is convex, so there's one minimum. L2 regularization keeps the weights from running off to infinity on linearly separable data

In plain terms. Points on a line: 1, 2, 10, 11; initial centers 1 and 2. Assign: point 1 goes to center 1, points 2, 10, 11 go to center 2. Recompute the centers: 1 and (2 + 10 + 11)/3 ≈ 7.67. Assign again: point 2 is now closer to center 1 (distance 1 vs 5.67). The new centers are 1.5 and 10.5, and nothing changes after that. The sum of squared distances only went down at every step: 145 → 48.7 → 17.6 → 1.

k-means (Lloyd's algorithm). Alternate: assign each point to the nearest center; recompute the centers as means. No step increases the sum of squared distances → the algorithm converges, but to a local minimum. Hence k-means++: the next center is chosen with probability proportional to the squared distance to the nearest center already chosen
PCA. Center the data; the principal components coincide with the right singular vectors of X (which are also the eigenvectors of the covariance, week F4). The share of explained variance is σᵢ² / Σσⱼ². Without centering, the first component points toward the mean rather than along the direction of spread

In plain terms. Four already-centered points: (2, 2), (−2, −2), (1, −1), (−1, 1). The sum of their squared lengths is 20. Project onto the direction (1, 1)/√2: the projections are 2√2, −2√2, 0, 0, the sum of their squares is 16 (the spread you keep), and 4 is left for the reconstruction error (the squared distance from a point to its projection). Onto the direction (1, 0): 10 kept, error 10. Always kept + error = 20: that's the Pythagorean theorem for each point. Keeping more and erring less turn out to be one problem, and (1, 1)/√2 wins in both senses.

PCA: two problems with one answer. The principal components simultaneously maximize the variance of the projection and minimize the reconstruction error, because the sum of these two quantities is constant. A linear autoencoder (the encoder h = W_e x compresses d numbers into k < d, the decoder x̂ = W_d h expands them back, trained with MSE) finds the same subspace as the first k PCA components, but the basis within it can be anything, not necessarily orthonormal. A nonlinear autoencoder generalizes this idea
k-means as a bridge to EM. A Lloyd step consists of two moves: guess the hidden part (the cluster of each point) and update the parameters (the centers) given that guess. EM (expectation–maximization, the algorithm for models with latent variables from F5) does the same thing softly: instead of "the point is in cluster 2" it says "cluster 2 with probability 0.7, cluster 1 with probability 0.3"
Decision tree. Greedily pick the feature and threshold that reduce impurity the most (Gini or entropy). Greed is needed because searching for the optimal tree is too expensive. Depth remains the main control against overfitting. Feature normalization isn't needed: thresholds are invariant to monotonic transformations

In plain terms. There are 10 positive examples. The model called only one example positive, and it really is positive: precision 1.0, recall 0.1. The arithmetic mean, 0.55, looks decent. F1 = 2·1.0·0.1 / (1.0 + 0.1) ≈ 0.18, close to the worse of the two metrics, so the recall failure can't be hidden.

Metrics. The confusion matrix (a "predicted × actual" table); precision (the share of correct ones among predicted positives), recall (the share found among all positives), F1. F1 is the harmonic mean, because it stays close to the smaller of the two and won't let one metric's failure hide. All three depend on the threshold

In plain terms. Positive examples got scores 0.9 and 0.4, negative ones 0.6 and 0.2. There are four "positive–negative" pairs: 0.9 > 0.6, 0.9 > 0.2, 0.4 > 0.2 are ordered correctly, 0.4 < 0.6 is not. AUC = 3/4 = 0.75. No threshold was needed for this, and if you square all the scores, the order and the AUC don't change.

ROC-AUC. ROC shows TPR (recall) as a function of FPR (the share of negatives wrongly called positive) as the threshold moves. **AUC equals the probability that a random positive example gets a higher score than a random negative one.** This interpretation immediately gives you a way to compute AUC via ranks, and shows that AUC doesn't depend on the threshold or on a monotonic transformation of the scores

Practice. Implement in NumPy: logistic regression with gradient descent; k-means with k-means++; PCA via SVD; a decision tree of depth ≤ 4 with Gini; functions for precision, recall, F1, and ROC-AUC two ways (via ranks and with trapezoids under the curve), matching to 1e-9. Check each implementation against sklearn on the same data.

If your process has a general ML round. Yandex, Sber and T-Bank give general ML a separate hour, and the questions there go wider than the five algorithms above. Reviewing four topics at the "explain it in a minute" level is enough; 10 self-check questions are in the rapid-fire section of the question bank, block "Classic ML".

In plain terms. A model gave probability 0.9 to ten emails, and 6 of them turned out to be spam. For this group the model promised 90% and hit 60%: it is overconfident. Calibration (agreement between predicted probability and observed frequency) is checked like this: bin the predictions by probability and in each bin compare the mean probability with the share of positives.

Calibration. Logistic regression is trained on log loss and is usually calibrated; boosting and networks trained for a long time are often overconfident. The fix is a monotone remapping of scores on a held-out set: Platt scaling (a logistic regression on top of the score) or isotonic regression. AUC doesn't change, while log loss and the Brier score (the mean squared difference between probability and outcome) improve. For neural networks and LLMs temperature scaling does the same job (dividing the logits by one fitted number)
Trees and ensembles. Bagging and random forests average deep trees trained on bootstrap samples and reduce variance. Gradient boosting builds shallow trees one after another, each fitted to the negative gradient of the loss of the current ensemble, and reduces bias. Its main knobs are the learning rate, the depth and the number of trees with early stopping
Splits and leakage. Data with time is split by time, data with repeated records from the same user is split by user, otherwise the test measures memorization. Leakage (a feature that isn't known yet at prediction time) shows up as suspiciously high quality and a single feature with huge importance
Ranking metrics. Classification scores each object on its own; ranking scores the order of the results. NDCG@k (the sum of relevances with a weight that falls with position, divided by the best possible sum) and MAP (the mean precision at the positions of relevant documents) punish a needed document at position 10 harder than at position 2. In RAG, recall@k and MRR play the same role (week 21 of the core)

Week outcomes

I can implement logistic regression, k-means, PCA and a depth ≤ 4 tree from an empty file, 20 minutes each.
I can derive the gradient of the logistic loss and show that its form matches the gradient of softmax + CE.
I can compute ROC-AUC via ranks and explain why it's the same thing as the area under the ROC curve.
I can pick a metric for a problem with class imbalance and justify the choice in a minute.
I can explain in a minute calibration, the difference between bagging and boosting, splitting by time, and NDCG.

Self-check

Why does k-means converge, and why not necessarily to the global minimum?
What happens to PCA if you forget to center the data?
What do ROC-AUC = 0.5 and ROC-AUC = 0.3 mean, and what do you do in the second case?

Resources: James, Witten, Hastie, Tibshirani, *An Introduction to Statistical Learning* (2nd ed., 2021; Python edition 2023), the chapters on classification, trees and unsupervised learning; Christopher Bishop, *Pattern Recognition and Machine Learning* (2006), for those who want deeper derivations; scikit-learn User Guide, the Model evaluation section, only to check the definitions of the metrics.

Exit check

You've finished the module when you've done everything on this list, without AI and without peeking:

Retake the diagnostic and score ≥ 24 out of 30, answering with derivations, not from memory of the keys.
In 45 minutes, from an empty file: logistic regression in NumPy with a hand-written gradient, a finite-difference gradient check and a ROC-AUC function.
In 5 minutes, out loud: the shapes of all tensors in the forward and backward pass of one linear layer with a batch.

If you didn't pass an item, go back to the matching F week instead of entering the core with a gap: week 1 of the core starts exactly where this module ends.