The Mathematics of the Desk Read

A Quant Trader's Textbook — the math behind fp-terminal, from first principles to shipping models

Preface

This book teaches the complete mathematics behind the fp-terminal Desk Read — the pre-open NQ game plan (gamma walls, the flip, the openBias probability, the expected-range model, the econ-surprise widening, and the accuracy tracker). It is written for the applied, self-taught quant: strong intuition and code, foundations built deliberately rather than assumed.

It is the companion text to the study program in ../quant-math-curriculum.md and the hands-on notebooks in ../../quant-lab/. The curriculum is the syllabus; this is the textbook; the notebooks are the lab.

Philosophy — derive, then build. Every result is derived by hand before it is used, and every chapter ends with exercises you run on your own NQ tape. Nothing is a black box by the end.

The organizing principle. Each chapter maps to one component the terminal actually prints, so you are never learning math in the abstract — you are learning the math that produces a specific number on your screen.

How to use this book

• Read a chapter, work its derivations on paper, then do its lab in quant-lab/.
• At ~1–3 hrs/week, budget roughly one chapter per month. The whole book is ~a year.
• Each chapter is self-contained but assumes the ones before it. Don't skip Part I even if it feels basic — the second-derivative idea in Ch. 2 is literally the same object as dealer gamma in Ch. 9.
• Solutions to all exercises are at the end of each chapter. Try first, then check.

Notation & conventions

Vectors are bold lowercase $\mathbf{x}$; matrices bold uppercase $\mathbf{X}$. "pts" = NQ index points ($\$20$ per point on the full-size contract).

Table of Contents

Part I — Foundations You'll Actually Use

• Ch. 1 — Probability & Distributions · the range line → openingRange.ts
• Ch. 2 — The Calculus of Sensitivity · why Γ is a 2nd derivative → the Greeks
• Ch. 3 — Statistics, Estimation & Significance · real edge vs noise → scoreBrief.ts
• Ch. 4 — Linear Algebra & Least Squares · every regression → the fitted models

Part II — Volatility & the Range Model

• Ch. 5 — Volatility: Implied, Realized & the √t Rule · → openingRange.ts
• Ch. 6 — Regression Done Right (and How It Lies) · → backtest-range.mjs

Part III — Options & Dealer Gamma: The Cage

• Ch. 7 — Random Walks → Brownian Motion → Itô · → the price model
• Ch. 8 — Black–Scholes & the Greeks · → option sensitivities
• Ch. 9 — Dealer Gamma Exposure (GEX): Building the Cage · → uw.ts levels

Part IV — Probability Models & the openBias

• Ch. 10 — Logistic Regression from Scratch · → openBias.ts
• Ch. 11 — Multivariate Factors & Multicollinearity · → the factor set

Part V — Inference, Backtesting & Edge

• Ch. 12 — Backtest Statistics & the Ways They Deceive · → the accuracy tracker
• Ch. 13 — Position Sizing, Expectancy & Kelly · → risk sizing
• Ch. 14 — Capstone: Ship a Validated Model · → extending the terminal

Appendices

• A — Matrix Calculus & Vector Derivatives · the machinery behind OLS, the logistic gradient & ridge
• B — The Gaussian Integral & Normal Facts · why the bell curve integrates to one, and the numbers you reuse
• C — Numerical Methods · finite differences, root-finding, Monte Carlo & gradient descent
• D — Notation & Symbols · the master symbol index and the book's recurring themes in one place

Chapters are individual files in this folder: ch01-...md through ch14-...md, plus appendix-...md.

Chapter 1 — Probability & Distributions

Maps to: the expected-range line (lib/openingRange.ts) and the coin-flip logic (lib/openBias.ts). Lab: quant-lab/module1.ipynb.

Learning objectives

By the end you can: (1) describe the opening move as a random variable with a distribution; (2) compute expectation and variance and say what each means for trading; (3) use the Normal and its 68–95–99.7 rule; (4) derive why price is lognormal while returns are normal, including the variance-drag term; (5) measure the fat tails of your own NQ opens and explain why they matter for stops.

1.1 Motivation: the range line is a distribution in disguise

When the Desk Read prints "5m ≈ 90pt," it is making a claim about a probability distribution — the distribution of how far NQ travels in the first five minutes. The single number is a summary; the risk you actually carry lives in the shape around it. A trader who reads "90pt" as a fact gets run over on the day the move is 250. A trader who reads it as "the center of a fat-tailed distribution" sizes for the tail. This chapter is the difference between those two traders.

1.2 Random variables

A random variable (RV) is a number whose value is determined by a random outcome. Before 9:30, the first-5-minute NQ move $X$ (in points) is an RV: you cannot know it, but you can describe its behavior.

RVs come in two flavors: - Discrete — countable outcomes (e.g., number of times price tags the call wall today). - Continuous — a continuum of outcomes (e.g., the move in points). NQ moves are effectively continuous.

1.3 Distributions: PDF and CDF

A continuous RV is described by its probability density function (PDF) $f(x)\ge 0$. The density is not a probability — for a continuous RV, $P(X=x)=0$. Probability is area under the density: $$P(a < X < b) = \int_a^b f(x)\,dx, \qquad \int_{-\infty}^{\infty} f(x)\,dx = 1.$$

The cumulative distribution function (CDF) accumulates that area: $$F(x) = P(X \le x) = \int_{-\infty}^{x} f(u)\,du.$$

Working rule. You plot the PDF (it's the histogram shape) but you compute with the CDF, because every probability question — "chance the move exceeds 100 pts?" — is a difference of CDF values: $P(X>c) = 1 - F(c)$.

1.4 Expectation and variance: the center and the spread

The expectation (mean) is the probability-weighted average — the long-run average value: $$\boxed{\ \mathbb{E}[X] = \int_{-\infty}^{\infty} x\,f(x)\,dx\ }$$

For the 5-minute directional move, $\mathbb{E}[X]\approx 0$: over many sessions the up-moves and down-moves cancel. This is the mathematical statement of your coin-flip logic — with no edge, the expected direction is zero, and only the spread is predictable.

The variance measures spread around the mean: $$\boxed{\ \text{Var}(X) = \mathbb{E}\!\left[(X-\mu)^2\right] = \mathbb{E}[X^2] - \mu^2\ }$$ and the standard deviation $\sigma = \sqrt{\text{Var}(X)}$ is the spread in the same units as $X$ (points). $\sigma$ is the quantity the range model estimates. Everything the terminal says about "how big today's open is" is a statement about $\sigma$.

Two properties you'll use constantly (for constants $a,b$ and independent $X,Y$): $$\mathbb{E}[aX+b]=a\,\mathbb{E}[X]+b, \qquad \text{Var}(aX+b)=a^2\text{Var}(X),$$ $$\text{Var}(X+Y)=\text{Var}(X)+\text{Var}(Y)\quad(\text{independent}).$$ That last one — variances add for independent pieces — is the seed of the √t rule in Chapter 5.

Worked example 1 (expectation & variance of a trade). A breakout play wins $+80$ pts with probability $0.45$ and loses $-60$ pts with probability $0.55$. Its expected P&L is $$\mathbb{E}[\text{P\&L}] = 0.45(80) + 0.55(-60) = 36 - 33 = +3\ \text{pts}.$$ Its variance is $\mathbb{E}[\text{P\&L}^2]-\mu^2 = \big[0.45(80^2)+0.55(60^2)\big] - 3^2 = 4860 - 9 = 4851$, so $\sigma \approx 69.6$ pts. The edge is $+3$ pts sitting inside a $\sim$70-pt swing — a whisper against the noise. That ratio, edge ÷ σ, is what position sizing is really about (Chapter 13).

1.5 The Normal distribution and the 68–95–99.7 rule

The Normal $\mathcal{N}(\mu,\sigma^2)$ has density $$f(x) = \frac{1}{\sigma\sqrt{2\pi}}\,\exp\!\left(-\frac{(x-\mu)^2}{2\sigma^2}\right).$$ Its indispensable fact:

Standardizing any Normal to the standard Normal $Z\sim\mathcal N(0,1)$ via $Z=(X-\mu)/\sigma$ lets you use one table/function $\Phi$ for everything: $$P(X > c) = 1 - \Phi\!\left(\frac{c-\mu}{\sigma}\right).$$

Worked example 2 (a tail probability). Suppose the opening move has $\mu=0,\ \sigma=45$ pts and is Normal. The chance of a move bigger than $100$ pts in either direction is $$P(|X|>100) = 2\big(1-\Phi(100/45)\big) = 2\big(1-\Phi(2.22)\big) = 2(1-0.9868) = 2.6\%.$$ So a $>100$-pt open "should" happen about once a month. Remember this number — §1.8 shows reality delivers it far more often.

Worked example 3 (a probability between two levels). With the same $\mu=0,\ \sigma=45$, what is the chance the 5-min close lands between $-30$ and $+60$ pts? Use the CDF as a difference: $$P(-30 < X < 60) = \Phi\!\left(\tfrac{60}{45}\right) - \Phi\!\left(\tfrac{-30}{45}\right) > = \Phi(1.33) - \Phi(-0.67) = 0.908 - 0.251 = 0.657 \approx 66\%.$$ Every "chance it lands in a band" question is just a subtraction of two CDF values — exactly how you'd size the odds of price holding inside the gamma cage.

1.6 Why returns are Normal but price is Lognormal — the key derivation

Prices compound multiplicatively: two +1% days give $1.01\times1.01$, not $+2\%$ exactly. Logarithms turn multiplication into addition, and a sum of many small independent shocks tends to a Normal (the Central Limit Theorem, Ch. 3). So we model the log-return as Normal: $$r = \ln\!\frac{S_T}{S_0} \sim \mathcal{N}(\mu,\sigma^2).$$

Then price is the exponential of a Normal: $$S_T = S_0\,e^{r}.$$ An RV of the form $e^{(\text{Normal})}$ is called lognormal. Two consequences:

(a) Price stays positive and is right-skewed. Since $e^r>0$ for all $r$, $S_T>0$ always — good, prices can't go negative. But the exponential stretches the upper tail more than the lower, so the price distribution is skewed right: an asset can go up 100% but only down 100% in the limit, asymmetrically.

(b) The average price is not $S_0 e^{\mu}$. Here is the result worth memorizing. Using the moment generating function of the Normal, $\mathbb{E}[e^{r}] = e^{\mu + \frac12\sigma^2}$, so $$\boxed{\ \mathbb{E}[S_T] = S_0\,e^{\mu + \frac{1}{2}\sigma^2}\ }$$

The extra $\tfrac12\sigma^2$ is the variance drag / convexity term. It appears because $e^x$ is convex, so by Jensen's inequality $\mathbb{E}[e^r] > e^{\mathbb{E}[r]}$ — averaging after exponentiating exceeds exponentiating the average.

Derivation of $\mathbb{E}[e^r]$ (do this by hand once): $$\mathbb{E}[e^{r}] = \int_{-\infty}^{\infty} e^{x}\,\frac{1}{\sigma\sqrt{2\pi}} e^{-\frac{(x-\mu)^2}{2\sigma^2}}\,dx.$$ Combine the exponents and complete the square in $x$: $$x - \frac{(x-\mu)^2}{2\sigma^2} = -\frac{1}{2\sigma^2}\Big(x - (\mu+\sigma^2)\Big)^2 + \mu + \tfrac12\sigma^2.$$ The remaining integral is a Normal density in $x$ (centered at $\mu+\sigma^2$) and integrates to 1, leaving $e^{\mu+\frac12\sigma^2}$. ∎

Why a trader cares. Volatility itself lifts the expected price while dragging the typical (median) path down. This $\tfrac12\sigma^2$ term is not a curiosity — it reappears as the $\tfrac12\sigma^2 S^2\Gamma$ term in the Black–Scholes equation (Ch. 8) and is the mathematical reason a delta-hedged option pays theta to own gamma. You have just derived, in one line, a term that runs through the entire options half of this book.

Interactive (rotate / zoom): the surface below is the lognormal price density as volatility $\sigma$ grows. Watch the peak slide left and the right tail stretch — that widening skew is the variance-drag term building.

1.7 A first look at conditional probability and Bayes (previewing openBias)

The openBias model answers a conditional question: given today's factors (VXN shift, overnight rates, overnight dollar), what is $P(\text{up in 5 min})$? The machinery: $$P(A \mid B) = \frac{P(A \cap B)}{P(B)}, \qquad \boxed{\ P(A\mid B) = \frac{P(B\mid A)\,P(A)}{P(B)}\ }\ \text{(Bayes)}.$$

Read Bayes as updating: start with a base rate $P(A)$ (the unconditional chance of an up-open, ≈50%), then tilt it by how much more likely the observed factors are under "up" vs "down." When the factors are equally likely under either outcome, the update does nothing and you are back to 50% — that is a coin-flip day, falling straight out of the math. We build this into a full model in Chapter 10.

Worked example 4 (a Bayes tilt — the openBias in miniature). Start from the base rate $P(\text{up})=0.50$. Suppose historically, overnight rates fell on $60\%$ of up-opens but only $40\%$ of down-opens. This morning rates are down. Update: $$P(\text{up}\mid \text{rates}\!\downarrow) = \frac{P(\downarrow\mid\text{up})\,P(\text{up})}{P(\downarrow\mid\text{up})P(\text{up}) + P(\downarrow\mid\text{down})P(\text{down})} > = \frac{0.60(0.50)}{0.60(0.50) + 0.40(0.50)} = \frac{0.30}{0.50} = 0.60.$$ One mildly-informative factor nudges $50\%\to60\%$. Stack several independent factors and the tilts compound into the openBias probability. But if a factor is equally likely under up and down ($0.50/0.50$), the fraction collapses back to $0.50$ — it contributes nothing, and if the factors disagree you land on the coin flip.

1.8 Fat tails: where the Normal lies to you

Real market returns are not Normal. They have excess kurtosis — a taller, thinner peak and fatter tails. Kurtosis is the standardized fourth moment; "excess" subtracts 3 so the Normal is the zero baseline: $$\text{ExcessKurt}(X) = \frac{\mathbb{E}[(X-\mu)^4]}{\sigma^4} - 3.$$ Equity-index opens routinely show excess kurtosis well above zero. Practically:

• The $>100$-pt open that the Normal said happens $2.6\%$ of the time (Worked example 2) happens materially more often. The fat-tail factor (empirical ÷ Normal) is typically $>1$, and it grows as you go further into the tail — the biggest days are far more common than Gaussian intuition admits.
• This is why the range model runs slightly wide (conservative) yet still gets blown through on the wildest opens — those live in the tail the Normal understates, not in a model error.
• And it is why we distrust small samples of a fat-tailed variable — the reasoning that stopped us "fixing" the range model off six volatile late-June sessions (Ch. 3 and Ch. 6 formalize this).

Worked example 5 (measuring the fat tail). Over 480 sessions, the $|5\text{-min move}|$ exceeded 100 pts on 38 of them, so empirically $P(|X|>100)=38/480=7.9\%$. The fitted Normal (Worked example 2) said $2.6\%$. The fat-tail factor is $7.9\%/2.6\% \approx 3.0\times$: reality delivers the "once-a-month" move about three times as often. Push the threshold to 150 pts and the factor grows further — this is exactly the number you compute on your own tape in the Module 1 lab.

Labs live in the Study app. The runnable fat-tail lab and its guided exercises for this chapter are in the Study app → Lab tab (◈ Study, linked at the top of the reader) — real Python on the NQ data snapshot.

1.9 Chapter summary

• An RV is described by its distribution (PDF for shape, CDF for probabilities).
• $\mathbb{E}[X]$ is the center (≈0 for the 5-min move → the coin flip); $\sigma$ is the spread (what the range model estimates).
• Variances add for independent pieces → seed of √t (Ch. 5).
• Normal: 68–95–99.7; standardize with $Z=(X-\mu)/\sigma$.
• Returns Normal ⇒ price Lognormal, with $\mathbb{E}[S_T]=S_0 e^{\mu+\frac12\sigma^2}$ — the variance-drag term that recurs in options.
• Real opens are fat-tailed: the Normal understates extremes; size risk for the tail.

Key formulas to carry forward:

Exercises

1.1 (Warm-up.) The 5-min move has $\sigma=40$ pts, $\mu=0$, assumed Normal. Find $P(X>60)$ and $P(|X|>80)$.

1.2 (Standardizing.) On a high-vol day $\sigma=70$ pts. Under the Normal, what move size is the "95% band" (the $\pm$ level containing 95% of opens)? How does that compare to the model's $m5$ output at a matching VIX?

1.3 (Variance drag.) Daily log-returns have $\mu=0$, $\sigma=1\%$. Using variance-additivity over 252 independent days, compute the annual variance drag $\tfrac12\sigma_{\text{yr}}^2$ and explain what it means for the gap between the average and the typical yearly outcome.

1.4 (Complete the square.) Fill in every step of the §1.6 derivation of $\mathbb{E}[e^r]=e^{\mu+\frac12\sigma^2}$.

1.5 (Lab — the milestone.) In module1.ipynb, compute $\sigma$ of your NQ 5-min move, then $P(|move|>100)$ empirically and under the fitted Normal. Report the fat-tail factor. Then re-run with thresholds 50, 100, 150 pts and describe how the factor changes.

1.6 (Interpretation.) Your stop is placed at the Normal "95% band" ($2\sigma$). Using your empirical fat-tail factor from 1.5, estimate how often price actually exceeds that stop versus the 2.5% the Normal promises. What does this say about sizing to Normal quantiles?

Solutions

Symbols & Greek letters in this chapter

Next: Chapter 2 — The Calculus of Sensitivity, where the second derivative you'll meet is the very same object as dealer gamma.

Chapter 2 — The Calculus of Sensitivity

Maps to: the option Greeks Δ and Γ that produce the gamma walls, flip, and cage (lib/uw.ts, computeGamma). Lab: compute Δ and Γ numerically (carried into Ch. 8's Black–Scholes).

Learning objectives

By the end you can: (1) read a derivative as a rate of change / slope; (2) recognize delta as a first derivative and gamma as a second derivative of option value; (3) use partial derivatives and the gradient to see each Greek as one sensitivity; (4) apply the Taylor expansion $dV\approx\Delta\,dS+\tfrac12\Gamma\,dS^2$; (5) compute derivatives numerically with finite differences; (6) explain long- vs short-gamma dealer behavior straight from the $\tfrac12\Gamma\,dS^2$ term.

2.1 Motivation: everything in options is a sensitivity

An option's price is a function of the underlying, time, and volatility. The Greeks are nothing but the sensitivities of that price — the rates at which it responds when each input moves. The gamma walls and the flip that the terminal draws are aggregate second-order sensitivities of dealer inventory. So to understand the cage, you first need the calculus of "how fast does value change when the market moves?" That is this chapter, and it is smaller than it sounds: two ideas — slope and curvature — carry the whole options half of the book.

2.2 The derivative: a rate of change

The derivative of a function $V(S)$ measures how fast $V$ changes per unit change in $S$ — the slope of its graph: $$\boxed{\ \frac{dV}{dS} = \lim_{h\to 0}\frac{V(S+h)-V(S)}{h}\ }$$ The limit says: take the average rate of change over a step $h$, then shrink $h$ to zero to get the instantaneous rate. Geometrically it is the slope of the tangent line touching the curve at $S$.

For an option, this first derivative with respect to the underlying is delta: $$\Delta = \frac{\partial V}{\partial S}.$$ $\Delta=0.55$ means the option gains about \$0.55 for each \$1 the underlying rises — the option's local "share-equivalent" exposure.

Worked example 1 (delta as a slope). A call is worth \$4.00 at $S=100$ and \$4.55 at $S=101$. Its delta is approximately the slope over that step: $$\Delta \approx \frac{4.55 - 4.00}{101 - 100} = 0.55.$$ The option behaves, locally, like 0.55 shares of the underlying.

2.3 The second derivative: curvature and gamma

Delta is not constant — it changes as $S$ moves. The rate at which delta changes is the second derivative of value, and it is gamma: $$\boxed{\ \Gamma = \frac{\partial \Delta}{\partial S} = \frac{\partial^2 V}{\partial S^2}\ }$$ Geometrically, the second derivative is curvature — how much the value curve bends away from its tangent line. A long option has $\Gamma>0$: the curve is convex (bends upward), so value beats the straight-line estimate whether the market rises or falls. That convexity is the whole appeal of being long options.

Worked example 2 (gamma as the change in delta). Suppose delta is $0.55$ at $S=100$ and $0.60$ at $S=101$. Then $$\Gamma \approx \frac{0.60 - 0.55}{101-100} = 0.05\ \text{per \$1}.$$ Every \$1 the underlying rises, the option picks up another $0.05$ of delta — it gets "longer" as it goes up. That self-reinforcing delta is exactly what dealers must hedge, and the source of the cage.

2.4 Many inputs: partial derivatives and the gradient

Option value depends on several things at once — price $S$, time $t$, volatility $\sigma$, rate $r$. A partial derivative is the rate of change with respect to one input while the others are held fixed, written with $\partial$ instead of $d$. Each Greek is a partial derivative of value: $$\Delta=\frac{\partial V}{\partial S},\quad \Gamma=\frac{\partial^2 V}{\partial S^2},\quad \Theta=\frac{\partial V}{\partial t},\quad \nu=\frac{\partial V}{\partial \sigma}.$$ The gradient $\nabla V$ collects all the first partials into one vector — the direction of steepest ascent, and the surface's slope in each input direction. Visualize a value surface over two inputs: the gradient is the tilt of the tangent plane, and each Greek is that plane's slope along one axis.

Interactive (rotate / zoom): the surface below is a smooth two-input value $f(x,y)$ with its tangent plane at the marked point. The plane's slope along $x$ is $\partial f/\partial x$ and along $y$ is $\partial f/\partial y$ — each option Greek is exactly one such directional slope.

2.5 The Taylor expansion — the master formula

How much does value change for a move $dS$? Taylor's expansion builds the answer from the derivatives: $$\boxed{\ V(S+dS) \approx V(S) + \Delta\,dS + \tfrac12\Gamma\,dS^2 + \Theta\,dt\ }$$ The first-order term $\Delta\,dS$ is the straight-line (tangent) estimate. The second-order term $\tfrac12\Gamma\,dS^2$ is the curvature correction — small for tiny moves, but it grows with the square of the move, so it dominates when the market jumps. $\Theta\,dt$ is the drift from time passing. This one line is the workhorse of the entire options book; you met it in Ch. 1 as the seed of dealer gamma.

Worked example 3 (delta + gamma approximation). With $\Delta=0.55$ and $\Gamma=0.05$, estimate the value change for a $+3$ move (ignore time): $$dV \approx \Delta\,dS + \tfrac12\Gamma\,dS^2 = 0.55(3) + \tfrac12(0.05)(3^2) = 1.65 + 0.225 = 1.875.$$ The delta term alone gives $1.65$; gamma adds $+0.225$ of curvature. For a $-3$ move the delta term flips to $-1.65$ but the gamma term stays $+0.225$ — gamma helps in both directions. That directionless positive is the essence of long gamma.

2.6 Derivatives without formulas: finite differences

You rarely need a closed form to get a Greek — you can compute it numerically from three price points. Using a small step $h$, the central finite differences are $$\Delta \approx \frac{V(S+h)-V(S-h)}{2h}, \qquad \Gamma \approx \frac{V(S+h)-2V(S)+V(S-h)}{h^2}.$$ The gamma formula is just "the change in the change": it compares the up-step and down-step gains. This is how you will read Δ and Γ off a Black–Scholes price in Chapter 8, and off a real chain in Chapter 9.

Worked example 4 (numeric delta and gamma). A call is worth $3.50,\ 4.00,\ 4.55$ at $S=99,100,101$ (so $h=1$). Then $$\Delta \approx \frac{4.55-3.50}{2} = 0.525, \qquad \Gamma \approx \frac{4.55 - 2(4.00) + 3.50}{1^2} = 0.05.$$ Three numbers, both Greeks — no formula required.

Labs live in the Study app. The runnable finite-difference Greeks lab and its guided exercises are in the Study app → Lab tab (◈ Study) — compute Δ and Γ on real curves there.

2.7 Why $\tfrac12\Gamma\,dS^2$ is the whole story of dealer gamma

A market maker who sells you an option delta-hedges: they hold $-\Delta$ of the underlying so their combined position has zero first-order exposure. With the $\Delta\,dS$ term cancelled by the hedge, the leftover P&L from a move is dominated by the curvature term: $$\text{hedged P\&L} \approx \tfrac12\Gamma\,dS^2.$$

• Long gamma ($\Gamma>0$): the hedged position gains $\tfrac12\Gamma\,dS^2>0$ for any move. To stay neutral the dealer must sell into rallies and buy into dips — supplying liquidity, dampening the tape and pulling price back toward the pin. This is the "cage" above the gamma flip.
• Short gamma ($\Gamma<0$): the position loses on any move, and re-hedging means buying rallies and selling dips — chasing the move, amplifying it. This is the accelerant below the flip.

The gamma flip is simply the price where aggregate dealer $\Gamma$ crosses zero — the boundary between the dampening and amplifying regimes. Chapter 9 aggregates $\Gamma$ across the whole option chain to locate it; here you have the mechanism in one term.

Worked example 5 (a long-gamma hedger's P&L). A dealer is long gamma with $\Gamma=0.05$ (per \$1, per contract-equivalent). If the underlying moves \$4 in either direction, the delta-hedged P&L is about $$\tfrac12\Gamma\,dS^2 = \tfrac12(0.05)(4^2) = 0.40.$$ They profit whether it rips or dumps — which is why they lean against the move (sell highs, buy lows) to lock it in. Multiply across thousands of contracts and that hedging is what pins price inside the cage.

2.8 Chapter summary

• A derivative is a rate of change — the slope of a curve; the limit of a difference quotient.
• Delta $=\partial V/\partial S$ is the first derivative; gamma $=\partial^2V/\partial S^2$ is the second (curvature, the change in delta).
• Each Greek is a partial derivative; the gradient collects them.
• Taylor: $dV\approx\Delta\,dS+\tfrac12\Gamma\,dS^2+\Theta\,dt$ — the master formula.
• Compute Greeks numerically with central finite differences.
• A delta-hedged position's P&L $\approx\tfrac12\Gamma\,dS^2$: long gamma dampens (cage), short gamma amplifies; the flip is where dealer $\Gamma=0$.

Key formulas to carry forward:

Exercises

2.1 (Slope.) A put is worth \$2.10 at $S=100$ and \$1.95 at $S=101$. Estimate its delta. Why is it negative?

2.2 (Curvature.) A call's delta is $0.40$ at $S=99$ and $0.52$ at $S=101$. Estimate its gamma per \$1.

2.3 (Taylor.) With $\Delta=0.50,\ \Gamma=0.04$, estimate the value change for moves of $+2$, $-2$, and $+6$. Which term dominates the big move, and why?

2.4 (Finite differences.) A call is worth $5.20,\ 5.70,\ 6.28$ at $S=149,150,151$. Compute $\Delta$ and $\Gamma$ by central differences.

2.5 (Dealer gamma.) A dealer is short gamma with $\Gamma=-0.03$. What is the hedged P&L for a \$5 move, and which way must they trade to re-hedge — with or against the move? What does that do to the tape?

2.6 (Concept.) In one sentence each, say what $\Delta$, $\Gamma$, and $\Theta$ measure, and which one is a second derivative.

Solutions

Symbols & Greek letters in this chapter

Next: Chapter 3 — Statistics, Estimation & Significance, the skill that tells a real edge from noise.

Chapter 3 — Statistics, Estimation & Significance

Maps to: the accuracy tracker (lib/scoreBrief.ts) — telling a real edge from noise. Lab: standard error, the √n rule, and why six sessions prove nothing.

Learning objectives

By the end you can: (1) treat a sample statistic as an estimator with its own sampling distribution; (2) compute the standard error and use the Central Limit Theorem; (3) build and read a confidence interval; (4) run a hypothesis test with a t-statistic and interpret a p-value; (5) compute the standard error of a proportion and the sample size needed to detect an edge; (6) recognize the traps — small samples of fat-tailed data, multiple testing — that make backtests lie.

3.1 Motivation: is the edge real, or is it noise?

The accuracy tracker grades every morning's read. Over a handful of sessions the numbers jump around wildly — one week the range model looks 71% too tight, the next it's dead on; a best-play goes 3-and-1, then 1-and-4. The single most valuable skill in quant trading is separating a real, repeatable edge from the noise of a small sample. That skill is this chapter. It is the discipline that stopped us from "fixing" the range model on six volatile late-June sessions (Chapter 1), and it is the line between a strategy and a curve-fit ghost.

3.2 Estimators and the sampling distribution

We rarely know the true population parameters — the real mean $\mu$ and standard deviation $\sigma$ of the opening move. We only have a sample: the sessions we've recorded. From the sample we compute estimates — the sample mean $\bar{x}=\frac{1}{n}\sum x_i$ and the sample standard deviation $s$ — that stand in for $\mu$ and $\sigma$.

Here is the pivotal idea. An estimator is a recipe applied to a random sample, so the estimator is itself a random variable. Draw a different sample and you'd get a different $\bar{x}$. The distribution of those possible values is the sampling distribution of the estimator. An estimator is unbiased if its sampling distribution is centered on the truth — the sample mean is unbiased: $\mathbb{E}[\bar{x}]=\mu$. Statistics is the study of that sampling distribution: how far the estimate is likely to sit from the truth.

3.3 The standard error and the Central Limit Theorem

How much does $\bar{x}$ bounce around the truth? Using variance-additivity from Chapter 1, for $n$ independent draws $\text{Var}(\bar{x})=\text{Var}\!\big(\tfrac1n\sum x_i\big)=\tfrac{1}{n^2}\sum\text{Var}(x_i)=\tfrac{\sigma^2}{n}$. Taking the square root gives the standard error of the mean: $$\boxed{\ \text{SE}(\bar{x}) = \frac{\sigma}{\sqrt{n}}\ }$$

Two consequences run through everything you'll do:

• The Central Limit Theorem (CLT): for large $n$, $\bar{x}$ is approximately Normal, $\mathcal{N}(\mu,\sigma^2/n)$, no matter how fat-tailed the individual move is. This is what lets us use the Normal machinery of Chapter 1 on the average, even though a single open is not Normal.
• The √n rule: precision improves only with the square root of the sample. To halve your uncertainty you need four times the data. This is why a handful of sessions tells you almost nothing.

Interactive (rotate / zoom): the surface is the density of the sample mean over sample size $n$. As $n$ grows the ridge narrows and rises — the estimate homing in on the truth at rate $1/\sqrt{n}$.

Worked example 1 (standard error). From $n=400$ sessions the sample standard deviation is $s=45$ pts. The standard error of the mean is $\text{SE}=45/\sqrt{400}=45/20=2.25$ pts. So while a single open swings by tens of points, the average open is pinned to within about $\pm4.5$ pts (two standard errors) of the truth.

3.4 Confidence intervals

The standard error turns into a range you can state. A 95% confidence interval for the mean is $$\boxed{\ \bar{x} \pm 1.96\cdot\text{SE}\ }$$ The 1.96 is the two-sigma point of the Normal (Chapter 1). Interpretation, carefully: if you repeated the whole sampling process many times, about 95% of the intervals you'd construct this way would contain the true $\mu$. It is a statement about the procedure's reliability, not a probability about this one interval.

Worked example 2 (a confidence interval for the drift). The 400-session sample mean move is $\bar{x}=+1.5$ pts with $\text{SE}=2.25$. The 95% CI is $1.5\pm1.96(2.25)=1.5\pm4.4$, i.e. $(-2.9,\ +5.9)$ pts. The interval straddles zero, so we cannot distinguish the average open from zero — the coin flip, made quantitative.

3.5 Hypothesis testing and the t-statistic

A hypothesis test formalizes "is it real?" You state a null hypothesis $H_0$ — the boring default, e.g. "the mean move is zero" or "the win rate is 50%." You then compute a test statistic that measures how far the data sit from the null, in standard errors: $$\boxed{\ t = \frac{\text{estimate}-\text{null}}{\text{SE}}\ }$$ Under $H_0$, $t$ is approximately standard Normal (Student's $t$ for small $n$ — slightly fatter-tailed). The p-value is the probability, if the null were true, of seeing a $t$ at least this extreme. If the p-value is below a chosen significance level $\alpha$ (commonly 0.05), you reject $H_0$. As a rule of thumb, $|t|>1.96$ means "significant at 5%."

One discipline that saves careers: failing to reject $H_0$ is not proving it. A coin-flip result means your data can't tell the difference yet — usually because $n$ is too small — not that the edge is exactly zero.

Worked example 3 (the coin-flip test). Test $H_0$: mean move $=0$. With $\bar{x}=1.5$ and $\text{SE}=2.25$, $t=1.5/2.25=0.67$. Since $|t|=0.67<1.96$, we fail to reject; the p-value is about 0.50. The opening drift is statistically indistinguishable from zero. That is the coin-flip open, proven — not assumed.

3.6 The standard error of a proportion — testing a win rate

Win rates are proportions, and they have their own standard error. For a win rate $\hat{p}$ from $n$ trades, $$\text{SE}(\hat{p}) = \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}, \qquad\text{and under } H_0:\ p=0.5,\quad \text{SE}=\sqrt{\frac{0.25}{n}}.$$ To test a win rate against a coin flip, $z=(\hat{p}-0.5)/\text{SE}$. Rearranging tells you the sample size to resolve an edge of size $d=\hat{p}-0.5$: roughly $n\approx 0.25\,(1.96/d)^2$. A 5-percentage-point edge needs a few hundred trades before you can trust it.

Worked example 4 (is a 55% win rate real?). A best-play wins 55% of $n=40$ triggered trades. Under $H_0:p=0.5$, $\text{SE}=\sqrt{0.25/40}=0.079$, so $z=(0.55-0.50)/0.079=0.63$ — not significant ($p\approx0.53$). Now take the same 55% over $n=400$: $\text{SE}=\sqrt{0.25/400}=0.025$, $z=0.05/0.025=2.0$ — now significant at ~5%. The edge didn't change; only at $n=400$ could you trust it. This is precisely what the accuracy tracker is quietly accumulating toward.

3.7 How backtests deceive

Two traps turn noise into false confidence — both formalized further in Chapter 12.

• Small samples of a fat-tailed variable. Chapter 1 showed opens are fat-tailed; averaging a handful of them gives a wildly uncertain estimate. Any statistic from six sessions carries an enormous confidence interval.
• Multiple testing / data-snooping. Test 20 strategies at the 5% level and, by chance alone, about one will look "significant." Search hard enough over a fixed history and you will find a beautiful backtest that means nothing. The fix is out-of-sample validation and honest accounting of how many things you tried.

Worked example 5 (why six sessions proved nothing). The range model looked 71% too tight over six late-June sessions. Suppose the session-to-session error has a standard deviation around 90%. Then the SE of a 6-session mean is $90/\sqrt{6}\approx37\%$, and a 95% CI on the observed $+71\%$ is roughly $71\pm2(37)=(-3\%,\,+145\%)$ — it comfortably includes zero. Six days cannot distinguish a broken model from a good one having a volatile week. That is exactly why we did not rebuild the model — and Chapter 6 confirmed it on 485 sessions.

3.8 Chapter summary

• An estimator is a random variable; its sampling distribution is what statistics studies.
• Standard error $=\sigma/\sqrt{n}$; the CLT makes the mean Normal; precision improves only like $\sqrt{n}$.
• 95% CI $=\bar{x}\pm1.96\,\text{SE}$ — a statement about the procedure.
• Hypothesis test: $t=(\text{estimate}-\text{null})/\text{SE}$; reject if the p-value $<\alpha$; failing to reject is not proof.
• Proportion SE $=\sqrt{\hat{p}(1-\hat{p})/n}$; a 5-point win-rate edge needs a few hundred trades.
• Small samples and multiple testing are how backtests lie — demand out-of-sample evidence.

Key formulas to carry forward:

Exercises

3.1 (Standard error.) A 100-session sample has $s=50$ pts. Find the standard error of the mean. How much data would you need to halve it?

3.2 (Confidence interval.) With $\bar{x}=-0.8$ pts and $\text{SE}=1.9$, give the 95% CI for the mean move. Is the drift distinguishable from zero?

3.3 (t-statistic.) A model's directional lean was correct on a 5-minute basis with mean signed move $\bar{x}=+2.0$ bps and $\text{SE}=3.0$ bps over its graded days. Compute $t$ and state the conclusion at 5%.

3.4 (Proportion.) A best-play is 12-8 (60% over 20 trades). Test it against a coin flip: compute $z$ and say whether it's significant.

3.5 (Sample size.) Using $n\approx0.25(1.96/d)^2$, how many trades to resolve a 4-percentage-point edge ($d=0.04$)? A 10-point edge?

3.6 (Concept.) In one sentence each: what does a p-value mean, and why is "failing to reject $H_0$" not the same as "proving $H_0$"?

Solutions

Symbols & Greek letters in this chapter

Next: Chapter 4 — Linear Algebra & Least Squares, the machinery behind every regression in the terminal.

Chapter 4 — Linear Algebra & Least Squares

Maps to: every fitted model — the range model a + b·VIX (lib/openingRange.ts, scripts/backtest-range.mjs) and the openBias coefficients (lib/openBias.ts). Lab: fit a line two ways (hand-coded normal equations vs. numpy) and check R².

Learning objectives

By the end you can: (1) read data as vectors and use the dot product; (2) state the least-squares problem geometrically; (3) derive the OLS line from scratch and reproduce the range model's coefficients; (4) write regression in matrix form — $\hat{\beta}=(X^\top X)^{-1}X^\top y$ — and see it as a projection; (5) compute and interpret R²; (6) extend a line to a plane for multiple predictors.

4.1 Motivation: every fitted number is least squares

The range model isn't magic — it's a straight line, $\text{range}\% \approx a + b\cdot\text{VIX}$, fit to hundreds of sessions. The openBias probability is a (logistic) regression on a handful of factors. Behind every "the model says" number in the terminal sits the same operation: find the coefficients that make the fit as close to the data as possible, where "close" means least squared error. This chapter builds that operation from vectors up, so those coefficients stop being a black box and become something you can derive, reproduce, and distrust intelligently.

4.2 Vectors and the dot product

A vector is an ordered list of numbers — a column of data. Stack every session's VIX into a vector $\mathbf{x}$ and every session's realized range into $\mathbf{y}$. Two operations do most of the work:

• Length (norm): $\|\mathbf{a}\| = \sqrt{a_1^2+a_2^2+\cdots} = \sqrt{\mathbf{a}\cdot\mathbf{a}}$.
• Dot product: $\mathbf{a}\cdot\mathbf{b} = \sum_i a_i b_i$. It measures alignment: large and positive when the vectors point the same way, zero when they are orthogonal (perpendicular). Correlation, projection, and least squares are all built on the dot product.

The single geometric fact to carry: two vectors are orthogonal exactly when their dot product is zero. That fact is the least-squares solution, as we'll see.

4.3 The least-squares problem

We want to fit $y \approx a + b\,x$. For each session the residual is what the line misses, $e_i = y_i - (a + b\,x_i)$. We choose $a,b$ to minimize the sum of squared residuals: $$\text{SSR}(a,b) = \sum_i \big(y_i - a - b\,x_i\big)^2.$$ Why squared? Squaring punishes big misses more, keeps the math differentiable, and — not coincidentally — makes the answer the one that treats the residuals as a vector and makes it as short as possible.

4.4 Deriving the OLS line

Minimize SSR by setting its two partial derivatives (Chapter 2!) to zero. Differentiating with respect to $a$ and $b$ and dividing out gives the normal equations, whose solution is $$\boxed{\ b = \frac{\sum_i (x_i-\bar{x})(y_i-\bar{y})}{\sum_i (x_i-\bar{x})^2}, \qquad a = \bar{y} - b\,\bar{x}\ }$$ Read the slope: it is the covariance of $x$ and $y$ divided by the variance of $x$ — how much $y$ moves per unit of $x$, scaled by how much $x$ itself varies. This is the exact formula that produced MODEL.m15 from 485 sessions of $(\text{VIX},\text{range}\%)$. Nothing more mysterious than a covariance over a variance.

Worked example 1 (the dot product). For $\mathbf{a}=(3,4)$ and $\mathbf{b}=(2,1)$: the dot product is $3(2)+4(1)=10$, and $\|\mathbf{a}\|=\sqrt{9+16}=5$. If instead $\mathbf{b}=(4,-3)$, then $\mathbf{a}\cdot\mathbf{b}=12-12=0$ — the vectors are orthogonal. Orthogonality = zero dot product is the hinge of everything that follows.

Worked example 2 (fit a line by hand). Four points: $(1,2),(2,2),(3,4),(4,5)$. Here $\bar{x}=2.5$, $\bar{y}=3.25$. Slope $b=\frac{\sum(x-\bar{x})(y-\bar{y})}{\sum(x-\bar{x})^2}=\frac{(-1.5)(-1.25)+(-0.5)(-1.25)+(0.5)(0.75)+(1.5)(1.75)}{2.25+0.25+0.25+2.25} > =\frac{5.0}{5.0}=1.0$, and $a=3.25-1.0(2.5)=0.75$. So $\hat{y}=0.75+1.0\,x$ — the closed form, by hand.

4.5 The matrix form — and why it's a projection

Line, plane, or a model with ten factors: it's all one equation. Stack a column of 1's (for the intercept) and your predictors into the design matrix $X$, put the outcomes in $y$, and the least-squares coefficients are $$\boxed{\ \hat{\beta} = (X^\top X)^{-1} X^\top y\ }$$ Here $X^\top$ is the transpose and $(\cdot)^{-1}$ the matrix inverse. The fitted values are $\hat{y}=X\hat{\beta}=X(X^\top X)^{-1}X^\top y$ — and that operator is a projection: it drops $y$ straight down onto the space of things your predictors can represent, and the residual $y-\hat{y}$ comes out orthogonal to that space. That orthogonality — residual dot predictors $=0$ — is the normal equations. Every regression in the terminal, simple or multiple, is this one projection.

Worked example 3 (reproduce the range model). Regressing realized 15-min range% on VIX over the sessions gives, via the boxed formula, a slope near $b\approx0.0293$ and intercept $a\approx-0.064$ — exactly the MODEL.m15 coefficients the terminal ships (Chapter 6 re-derives them on 485 sessions). The model is a two-number least-squares fit; you can reproduce it in the lab.

4.6 How good is the fit? R²

Split the outcome's variation into what the fit explains and what it leaves as residual. With $\text{SST}=\sum(y_i-\bar{y})^2$ (total) and $\text{SSR}=\sum(y_i-\hat{y}_i)^2$ (residual), $$\boxed{\ R^2 = 1 - \frac{\text{SSR}}{\text{SST}}\ }$$ $R^2$ is the fraction of $y$'s variance the model captures — 1 is perfect, 0 is no better than the mean. But a warning you'll formalize in Chapter 6: $R^2$ never decreases when you add predictors, so a high $R^2$ is easy to fake by piling on variables. It measures fit in-sample, not predictive power.

Worked example 4 (compute R²). From Worked example 2, $\hat{y}=0.75+x$ gives fitted values $1.75,2.75,3.75,4.75$ and residuals $0.25,-0.75,0.25,0.25$. So $\text{SSR}=0.0625+0.5625+0.0625+0.0625=0.75$, while $\text{SST}=\sum(y-\bar{y})^2=(-1.25)^2+(-1.25)^2+(0.75)^2+(1.75)^2=5.0$. Thus $R^2=1-0.75/5.0=0.85$ — the line explains 85% of the variation.

4.7 From line to plane: multiple regression

Add a second predictor and the "line" becomes a plane in three dimensions: $\hat{y}=\beta_0+\beta_1 x_1+\beta_2 x_2$. The matrix formula $\hat{\beta}=(X^\top X)^{-1}X^\top y$ is unchanged — $X$ just gains a column. This is how the openBias model combines several factors at once, and it's the setting for Chapter 11, where correlated predictors (multicollinearity) start to cause trouble. Geometrically, least squares now finds the flat plane closest to a cloud of points in space.

Interactive (rotate / zoom): points scattered around a plane, with the least-squares plane fit through them. The residuals are the vertical gaps; least squares makes their squared total as small as possible.

Worked example 5 (a plane by the normal equations). With two predictors, $X$ has three columns $[\,\mathbf{1}\ \ \mathbf{x_1}\ \ \mathbf{x_2}\,]$. Form the $3\times3$ matrix $X^\top X$ and the vector $X^\top y$, then solve $\hat{\beta}=(X^\top X)^{-1}X^\top y$. By hand it's tedious; in the lab it's three lines of numpy — and it's the same operation whether you have two predictors or twenty.

4.8 Chapter summary

• Data are vectors; the dot product measures alignment, and zero dot product = orthogonal.
• Least squares minimizes the sum of squared residuals — the line/plane closest to the data.
• Simple OLS: $b=\text{cov}(x,y)/\text{var}(x)$, $a=\bar{y}-b\bar{x}$ — this is the range model.
• Matrix form $\hat{\beta}=(X^\top X)^{-1}X^\top y$ works for any number of predictors; it's a projection, and the residual is orthogonal to the fit.
• R² $=1-\text{SSR}/\text{SST}$ is in-sample fit — and it never drops when you add predictors (Ch. 6 warning).

Key formulas to carry forward:

Exercises

4.1 (Dot product.) For $\mathbf{a}=(2,-1,2)$ and $\mathbf{b}=(1,2,2)$, compute $\mathbf{a}\cdot\mathbf{b}$ and $\|\mathbf{a}\|$. Are they orthogonal?

4.2 (Slope.) Points $(0,1),(1,3),(2,3),(3,5)$. Compute $\bar{x},\bar{y}$, then the OLS slope $b$ and intercept $a$.

4.3 (Interpretation.) The range model has slope $b\approx0.0293$ (range% per VIX point). In words, what does that slope say happens to the expected range as VIX rises by 5 points?

4.4 (R².) A fit leaves $\text{SSR}=12$ out of $\text{SST}=80$. What is $R^2$? What fraction of variance is unexplained?

4.5 (Matrix shapes.) You regress range% on VIX and a data-surprise factor over 400 sessions. Give the shapes of $X$, $X^\top X$, and $\hat{\beta}$.

4.6 (Concept.) Why does the least-squares residual come out orthogonal to the predictors, and what equation does that orthogonality give you?

Solutions

Symbols & Greek letters in this chapter

Next: Chapter 5 — Volatility: Implied, Realized & the √t Rule, where σ meets the clock.

Chapter 5 — Volatility: Implied, Realized & the √t Rule

Maps to: the expected-range model's VIX- and time-scaling (lib/openingRange.ts, scripts/backtest-range.mjs). Lab: watch a random walk spread like √t, and turn VIX into a per-minute σ.

Learning objectives

By the end you can: (1) distinguish realized from implied volatility and say what VIX measures; (2) annualize and de-annualize a volatility; (3) derive the √t rule from variance-additivity; (4) turn a VIX level into a per-minute σ and an expected opening move; (5) explain why the fitted range model is √t-flavored but not pure √t; (6) recognize volatility clustering as why VIX is informative at all.

5.1 Motivation: the range line has two knobs

The expected-range model scales with two things: the level of VIX, and the length of the window — 1-minute, 5-minute, 15-minute. Both come from a single idea: volatility, measured per unit of time, and the rule for how it grows as time passes. This chapter derives that growth (the √t rule) and shows how the annual number on the VIX tape becomes the per-minute σ that sets your opening range. Chapter 4 gave you the fitting; this chapter gives you the physics the fit is approximating.

5.2 Realized volatility

Realized volatility is measured, backward-looking: take a series of returns, compute their standard deviation, and scale it to a standard horizon. If minute returns have standard deviation $\sigma_{\text{min}}$, the realized vol is that $\sigma$ — it's just Chapter 1's spread, applied to returns. It answers "how much did the market actually move?"

5.3 Implied volatility and VIX

Implied volatility is forward-looking: it's the σ the market has priced into options right now. Because an option is worth more when future moves are expected to be larger, you can invert an option's price to back out the volatility it implies. VIX aggregates the implied vols across a strip of S&P 500 options into a single number: the market's expected annualized volatility over the next 30 days, quoted in percent. (Its Nasdaq-100 cousin is VXN, the more relevant gauge for NQ.) Two facts to keep:

• VIX is annualized and in percent — a VIX of 18 means "≈18% annualized σ."
• Implied usually sits a little above realized — the volatility risk premium: option sellers charge for bearing the risk of a surprise. That gap is why selling options tends to pay, and why it occasionally blows up.

5.4 The √t rule — why range grows with the square root of time

Here is the engine. From Chapter 1, variances add over independent periods. If each minute's return has variance $\sigma^2$, then over $t$ independent minutes the total variance is $t\,\sigma^2$, so the standard deviation over $t$ minutes is $$\boxed{\ \sigma_{[t]} = \sigma\sqrt{t}\ }$$ Volatility scales with the square root of time. Double the horizon and the typical move grows by $\sqrt{2}$, not by 2. This one rule ties every horizon together and lets you move a single volatility number between timescales.

De-annualizing VIX. A year of trading is about 252 days, and a regular-hours session about 390 minutes. So to turn the annual VIX into a per-minute σ, divide by the square root of the number of periods: $$\sigma_{\text{day}} = \frac{\text{VIX}/100}{\sqrt{252}}, \qquad \sigma_{\text{min}} = \frac{\sigma_{\text{day}}}{\sqrt{390}} = \frac{\text{VIX}/100}{\sqrt{252\cdot390}}.$$

Interactive (rotate / zoom): expected opening range as a surface over VIX and horizon — rising roughly linearly in VIX and like $\sqrt{t}$ in the window length. That surface is the range model, generalized.

Worked example 1 (realized vol, annualized). Minute returns over a session have standard deviation $\sigma_{\text{min}}=0.03\%$. Annualize with the √t rule: multiply by $\sqrt{252\cdot390}=\sqrt{98280}\approx313.5$. So $\sigma_{\text{ann}}\approx0.03\%\times313.5\approx9.4\%$ — a calm session's realized vol.

Worked example 2 (VIX to a per-minute σ). With VIX $=18$: $\sigma_{\text{day}}=0.18/\sqrt{252}=1.13\%$, and $\sigma_{\text{min}}=1.13\%/\sqrt{390}=0.0574\%$. At NQ $=30{,}000$ that per-minute σ is about $17$ points — the typical size of a single one-minute candle when VIX is 18.

5.5 From σ to the expected range

A 1-σ move over $t$ minutes is $\sigma_{\text{min}}\sqrt{t}$; the range (high minus low) is a bit larger than a single 1-σ move, so the expected opening range is a constant times $\sigma\sqrt{t}$: $$\text{range over } N \text{ min} \approx c\cdot\sigma_{\text{min}}\sqrt{N}\cdot S.$$ That is the skeleton of the model. The empirical fit of Chapter 4, $\text{range}\% \approx a + b\cdot\text{VIX}$, is this relationship with the constants pinned by data — the slope $b$ absorbs the $c$ and the σ-per-VIX conversion, and the intercept $a$ patches the places pure theory misses.

Worked example 3 (expected 5-minute move). From Worked example 2, $\sigma_{\text{min}}=0.0574\%$. A 5-minute 1-σ move is $0.0574\%\times\sqrt{5}=0.128\%$; at NQ $30{,}000$ that's about $38$ points. The model's $m5$ near $90$ points is the range (a ~2–2.5-σ high-low span, on a fatter-than-Normal open) — same physics, wider because range exceeds a single σ and the open is fat-tailed (Chapter 1).

5.6 Why the fit isn't pure √t

If range were exactly $c\,\sigma\sqrt{t}$, then $m15/m1$ would be $\sqrt{15}\approx3.9$. In the fitted model it's closer to $135/49\approx2.8$ — flatter. Three reasons the open bends the √t line:

• The opening auction front-loads volatility: the first minute is unusually large, lifting $m1$ and shrinking the ratio.
• Mean-reversion and microstructure damp the later minutes relative to a pure random walk.
• The intercept $a$ in the fit is a floor that adds a fixed amount at every horizon, further flattening the ratio. The theory sets the shape; the data supplies the corrections.

Worked example 4 (the √t ratio). Pure √t predicts the 15-min range is $\sqrt{15}/\sqrt{1}=3.87\times$ the 1-min range. The model's $135/49=2.76$ is smaller — the gap is exactly the opening-auction front-loading plus the intercept. When you see a 1-minute range near the full 5-minute range, that's the auction, not a broken model.

5.7 Volatility clustering — why VIX predicts anything

Volatility is persistent: big days cluster with big days, calm with calm. Today's realized vol is the best cheap predictor of tomorrow's — the intuition behind GARCH models, and the reason a level like VIX carries real information about the coming session rather than being noise. It's also why a single quiet or wild open doesn't reset the regime; you weight recent volatility, not just the last print. Chapter 6 puts this fitting on rigorous footing.

Worked example 5 (does the model's √t hold up?). In the lab you'll simulate thousands of random-walk paths and measure their spread at each step: on a log-log plot the slope comes out ≈ 0.5, confirming $\sigma\propto\sqrt{t}$ from first principles — the exact scaling the range model leans on, verified on data you generate.

5.8 Chapter summary

• Realized vol is measured from returns; implied vol (VIX/VXN) is priced into options and runs a bit above realized (the vol risk premium).
• √t rule: variances add, so $\sigma_{[t]}=\sigma\sqrt{t}$ — volatility scales with the square root of time.
• De-annualize VIX with $\sqrt{252\cdot390}$ to get a per-minute σ; the expected range is $\approx c\,\sigma\sqrt{N}\,S$.
• The fitted $a+b\cdot\text{VIX}$ is this physics with data-pinned constants; the open bends pure √t (auction, mean-reversion, intercept).
• Volatility clusters, which is why VIX is informative.

Key formulas to carry forward:

Exercises

5.1 (Annualize.) Minute returns have $\sigma_{\text{min}}=0.02\%$. Annualize it (use $\sqrt{252\cdot390}\approx313.5$).

5.2 (De-annualize.) VIX $=24$. Find $\sigma_{\text{day}}$ and $\sigma_{\text{min}}$ (percent).

5.3 (√t.) If the expected 1-minute range is 50 pts, what does pure √t predict for the 9-minute range?

5.4 (Per-minute move.) Using $\sigma_{\text{min}}$ from 5.2 and NQ $=30{,}000$, what is a typical 1-minute move in points?

5.5 (Ratio.) The model gives $m1=53$, $m15=153$. What is the observed $m15/m1$ ratio, and how does it compare to pure √t? What accounts for the difference?

5.6 (Concept.) Why does doubling the time horizon multiply the expected move by $\sqrt{2}$ rather than 2?

Solutions

Symbols & Greek letters in this chapter

Next: Chapter 6 — Regression Done Right (and How It Lies), where we fit — and stress-test — the range model.

Chapter 6 — Regression Done Right (and How It Lies)

Maps to: the range-model recalibration (scripts/backtest-range.mjs) and every fitted model's trust. Lab: watch overfitting happen — train error falls while test error U-turns.

Learning objectives

By the end you can: (1) state the OLS assumptions and what breaks when each fails; (2) read residual plots for trouble; (3) explain why R² never decreases and use adjusted R²; (4) diagnose overfitting via the bias–variance tradeoff; (5) validate a model out-of-sample (train/test, cross-validation, walk-forward); (6) recount the range-model case study — why 485 sessions kept the coefficients and 6 sessions changed nothing.

6.1 Motivation: fitting is easy; trusting is hard

Chapter 4 made it three lines of numpy to fit a line. That ease is the danger. A regression will always return coefficients and an R², whether or not there's any real relationship — and it will happily fit noise if you let it. This chapter is the discipline that separates a model you can trade from a curve-fit mirage. It's the exact reasoning that recalibrated the range model on 485 sessions and, on the flip side, refused to "fix" it on six.

6.2 The OLS assumptions

Least squares gives the best line under a handful of assumptions; knowing them tells you when to distrust the fit:

• Linearity — the relationship really is a straight line (or plane). Curvature ⇒ the line is biased.
• Independence — observations don't lean on each other. Overlapping windows or autocorrelated days break this and make errors look smaller than they are.
• Homoscedasticity — constant residual variance. If the scatter fans out (bigger errors at high VIX), your standard errors are wrong.
• No perfect collinearity — predictors aren't redundant copies (Chapter 11).
• (For inference) roughly Normal residuals — needed for the p-values and confidence intervals of Chapter 3.

6.3 Residual diagnostics — let the errors talk

The single most useful check: plot the residuals $e_i=y_i-\hat{y}_i$ against the fitted values. A healthy fit leaves a structureless cloud — no trend, no fan, no curve. Any pattern is the model telling you an assumption is violated: a smile means missing curvature; a funnel means heteroscedasticity; a lone point far out is an outlier worth investigating.

6.4 R² revisited — and why it lies

Chapter 4 warned it; here's the mechanism. R² can only rise (never fall) when you add a predictor, because the extra column gives least squares more freedom to reduce the in-sample residual — even if the predictor is pure junk. So a high R² proves nothing on its own. Two guards:

• Adjusted R² penalizes extra parameters: $\ \bar{R}^2 = 1-(1-R^2)\dfrac{n-1}{n-p-1}\ $, which can fall when a useless predictor is added.
• The real test is out-of-sample R² — how well the model explains data it never saw.

Worked example 1 (adjusted R² catches junk). Fit $y$ on a real predictor: $R^2=0.80$ with $n=40,\ p=1$, so $\bar{R}^2=1-0.20\cdot\frac{39}{38}=0.795$. Add a random junk predictor and $R^2$ ticks up to, say, $0.805$; but $\bar{R}^2=1-0.195\cdot\frac{39}{37}=0.795$ — flat or down. Plain R² rewarded the junk; adjusted R² didn't.

6.5 Overfitting and the bias–variance tradeoff

Give a model enough flexibility and it stops learning the signal and starts memorizing the noise. Total expected prediction error decomposes into three pieces: $$\boxed{\ \mathbb{E}[\text{test error}] = \underbrace{\text{bias}^2}_{\text{too rigid}} + \underbrace{\text{variance}}_{\text{too flexible}} + \underbrace{\text{noise}}_{\text{irreducible}}\ }$$ A too-simple model has high bias (it can't capture the shape); a too-complex model has high variance (it swings wildly with the sample). The sweet spot is in between — and you cannot find it from in-sample fit, because in-sample error only ever falls with complexity.

Interactive (rotate / zoom): out-of-sample error over model complexity and training size. Error is worst where complexity is high and data is scarce — the overfitting corner — and the safe region grows as $n$ increases.

Worked example 2 (in-sample vs out-of-sample). Fit a degree-12 polynomial to 25 noisy points: train $R^2$ comes out near 1 (it threads every point), but test $R^2$ on fresh points is poor — often negative, worse than just predicting the mean. The training fit was memorization, not learning. You'll reproduce this in the lab.

6.6 Out-of-sample validation — the only honest test

Because in-sample metrics can be fooled, you must judge a model on data it did not fit:

• Train/test split — fit on one slice, score on another.
• Cross-validation — rotate the held-out slice so every point is tested once; average the scores.
• Walk-forward — for time series, always train on the past and test on the future, mimicking live use. This is the only split that respects the arrow of time and avoids look-ahead bias.

Worked example 3 (why +71% wasn't evidence). The six-session "+71% too tight" was an in-sample average over a tiny, volatile window. The 485-session recalibration was the real test: full-sample, it reproduced the coefficients (they were fine); a recent-120 re-fit collapsed to $r\approx0.11$ — pure overfitting of a short noisy window. Out-of-sample discipline said "leave it," and that's what we did.

6.7 Regularization — buying stability with a little bias

When flexibility is unavoidable, regularization tames the variance by shrinking the coefficients. Ridge regression adds a penalty on their size: $$\min_\beta\ \underbrace{\sum_i (y_i - X_i\beta)^2}_{\text{fit}} + \lambda\underbrace{\sum_j \beta_j^2}_{\text{penalty}}.$$ The knob $\lambda$ trades a touch of bias for a large drop in variance — often improving out-of-sample error. It's the principled version of "don't let any one coefficient run wild on thin evidence."

Worked example 4 (spurious R²). Regress pure noise on 10 unrelated noise predictors with $n=30$. You'll get an R² well above zero — the model "explains" a third of the variance of nothing, because 10 free knobs can chase random wiggles. In-sample R² without out-of-sample proof is not evidence; it's decoration.

6.8 Chapter summary

• OLS rests on linearity, independence, homoscedasticity, no collinearity (+ Normal residuals for inference).
• Read residual plots — patterns reveal violated assumptions.
• R² only rises with more predictors; use adjusted R² and, above all, out-of-sample R².
• Bias–variance: in-sample error falls with complexity, test error is U-shaped; overfitting = fitting noise.
• Validate out-of-sample (train/test, cross-validation, walk-forward); ridge trades bias for stability.
• The range model survived 485-session validation; six sessions never had the power to change it.

Key formulas to carry forward:

Exercises

6.1 (Adjusted R².) A model has $R^2=0.62$ with $n=50$ and $p=3$ predictors. Compute $\bar{R}^2$.

6.2 (Residuals.) You plot residuals vs fitted and see a clear upward-opening funnel (small errors on the left, large on the right). Which assumption is violated, and what does it break?

6.3 (Overfitting.) Train error keeps falling as you add polynomial degree, but test error bottomed out at degree 3 and rises after. Which degree do you ship, and why?

6.4 (Concept.) Explain in one sentence why in-sample R² cannot, by itself, tell you a model will predict well.

6.5 (Validation.) Why must a time-series backtest use walk-forward (train past → test future) rather than a random train/test split?

6.6 (Bias–variance.) Classify each as high-bias or high-variance: (a) a horizontal line fit to a clear upward trend; (b) a degree-15 polynomial through 15 noisy points.

Solutions

Symbols & terms in this chapter

Next: Chapter 7 — Random Walks → Brownian Motion → Itô, the stochastic model prices are built on.

Chapter 7 — Random Walks → Brownian Motion → Itô

Maps to: the stochastic model prices are built on — the foundation of Black–Scholes (Ch. 8) and dealer gamma (Ch. 9). Lab: simulate GBM paths and price a call by Monte Carlo.

Learning objectives

By the end you can: (1) build a random walk and see its variance grow with time; (2) take the limit to Brownian motion and use $W(t)\sim\mathcal N(0,t)$; (3) write the geometric Brownian motion (GBM) SDE for prices; (4) apply Itô's lemma informally and see the $\tfrac12\sigma^2 S^2 V_{SS}$ term appear because $(dW)^2=dt$; (5) price a payoff by Monte Carlo; (6) connect all of it back to gamma.

7.1 Motivation: you need a model of how prices move

To price an option (Chapter 8) or understand why dealer hedging pins the tape (Chapter 9), you first need a mathematical model of price motion. That model is the random walk and its continuous-time limit, Brownian motion, dressed up for prices as geometric Brownian motion. And the tool for doing calculus on it, Itô's lemma, turns out to be Chapter 2's Taylor expansion with one extra rule — a rule that resurrects the gamma term you already know. This chapter is short on new intuition and long on payoff.

7.2 The random walk

Take independent steps $\pm 1$ and add them up: $S_n = \xi_1+\xi_2+\cdots+\xi_n$. Two facts, both from Chapter 1: the mean stays at $0$ (each step is fair), and — because variances add — the variance after $n$ steps is $n$, so the standard deviation is $\sqrt{n}$. That's the √t rule of Chapter 5, in its rawest form: the walk drifts nowhere on average but spreads like the square root of time.

7.3 Brownian motion

Shrink the step size and speed up the clock in the right proportion and the random walk converges to Brownian motion (the Wiener process) $W(t)$ — the continuous-time model of pure randomness. Its defining properties:

• Starts at zero: $W(0)=0$, and paths are continuous.
• Independent increments: non-overlapping pieces are independent.
• Gaussian: $W(t)\sim\mathcal N(0,t)$ — variance equals the elapsed time, so $\text{std}=\sqrt{t}$.

We write an infinitesimal increment as $dW$, with $\mathbb{E}[dW]=0$ and $\text{Var}(dW)=dt$. The single rule that makes stochastic calculus different from ordinary calculus: $$\boxed{\ (dW)^2 = dt\ }$$ An ordinary $(dt)^2$ is negligible, but $(dW)^2$ is not — it's of order $dt$. Hold onto this; it's the whole twist.

7.4 Geometric Brownian motion — the price model

Prices are positive and move multiplicatively (a 1% move means the same thing at any price), so we don't let the price itself be Brownian — we let its logarithm be. The resulting stochastic differential equation is geometric Brownian motion: $$\boxed{\ dS = \mu S\,dt + \sigma S\,dW\ }$$ The $\mu S\,dt$ term is the drift (steady growth); the $\sigma S\,dW$ term is the diffusion (random shocks scaled by volatility). Solving it gives $S_T = S_0\,e^{(\mu-\frac12\sigma^2)T + \sigma W(T)}$ — the price is lognormal (Chapter 1), complete with the $-\tfrac12\sigma^2$ that is the flip side of the variance drag.

Interactive (rotate / zoom): the distribution of a GBM price as it evolves forward in time — a lognormal ridge that spreads and skews the further out you look. That widening cone is the uncertainty an option must be priced against.

7.5 Itô's lemma — where gamma comes back

Suppose an option's value is $V(S,t)$. How does $V$ change over $dt$? Taylor-expand (Chapter 2), keeping terms up to second order in $dS$: $$dV \approx \frac{\partial V}{\partial t}dt + \frac{\partial V}{\partial S}dS + \tfrac12\frac{\partial^2 V}{\partial S^2}(dS)^2.$$ Now substitute $dS=\mu S\,dt+\sigma S\,dW$ and, crucially, use $(dW)^2=dt$ (and drop the true $(dt)^2$ terms). The $(dS)^2$ term becomes $\sigma^2 S^2 (dW)^2 = \sigma^2 S^2\,dt$ — it survives. Collecting terms gives Itô's lemma: $$\boxed{\ dV = \Big(\frac{\partial V}{\partial t} + \mu S\frac{\partial V}{\partial S} + \tfrac12\sigma^2 S^2\frac{\partial^2 V}{\partial S^2}\Big)dt + \sigma S\frac{\partial V}{\partial S}\,dW\ }$$

Look at the term $\tfrac12\sigma^2 S^2\,\dfrac{\partial^2 V}{\partial S^2}$. That second derivative is gamma — the very curvature term from Chapter 2's $\tfrac12\Gamma\,dS^2$. It appears here for exactly the same reason it mattered for dealer hedging: convexity times the size of the move squared. Ordinary calculus would have thrown it away; the rule $(dW)^2=dt$ keeps it, and it becomes the engine of option pricing. This is the through-line of the whole book meeting itself.

Worked example 1 (random-walk spread). After $n=100$ fair $\pm1$ steps, the variance is $100$ and the standard deviation is $\sqrt{100}=10$. After $400$ steps it's $\sqrt{400}=20$ — quadruple the time, double the spread. Pure √t.

Worked example 2 (Itô on $V=S^2$). Take the simple function $V=S^2$, so $V_S=2S$, $V_{SS}=2$, $V_t=0$. Itô gives $dV=(2\mu S^2 + \tfrac12\sigma^2 S^2\cdot 2)\,dt + 2\sigma S^2\,dW = (2\mu+\sigma^2)S^2\,dt + 2\sigma S^2\,dW$. Ordinary calculus would have said $dV=2S\,dS$ and missed the extra $\sigma^2 S^2\,dt$ — that surviving term is the $(dW)^2=dt$ effect, in miniature.

7.6 Monte Carlo — pricing by simulation

If you can simulate the paths, you can price almost anything: generate many GBM terminal prices, compute the payoff on each, average, and discount: $$\text{price} \approx e^{-rT}\cdot\frac{1}{N}\sum_{i=1}^{N}\text{payoff}(S_T^{(i)}).$$ By the Central Limit Theorem (Chapter 3) the estimate is Normal around the true value with a standard error that shrinks like $1/\sqrt{N}$ — quadruple the paths to halve the error. In the lab you'll price a call this way and watch it converge; in Chapter 8 you'll get the same number from a formula.

Worked example 3 (Monte Carlo error). A Monte Carlo call price from $N=10{,}000$ paths has some standard error $\varepsilon$. To halve it you need $4\times$ the paths ($40{,}000$) — the $1/\sqrt{N}$ rule again. Precision in simulation obeys the same √-law as everything else in this book.

7.7 Chapter summary

• A random walk drifts nowhere but spreads like $\sqrt{t}$ (variances add).
• Its limit is Brownian motion: $W(t)\sim\mathcal N(0,t)$, with the key rule $(dW)^2=dt$.
• Prices follow GBM: $dS=\mu S\,dt+\sigma S\,dW$, giving lognormal prices.
• Itô's lemma is Taylor plus $(dW)^2=dt$; it revives the $\tfrac12\sigma^2 S^2 V_{SS}$ = gamma term.
• Monte Carlo prices any payoff by averaging simulated outcomes, with $1/\sqrt{N}$ error.

Key formulas to carry forward:

Exercises

7.1 (Random walk.) After 900 fair $\pm1$ steps, what are the mean and standard deviation of the position?

7.2 (Brownian.) $W(4)\sim\mathcal N(0,4)$. What is its standard deviation, and (roughly) the chance $|W(4)|>4$?

7.3 (The key rule.) Why does the $(dS)^2$ term survive in Itô's lemma while it vanishes in ordinary calculus?

7.4 (Itô.) Apply Itô's lemma to $V=\ln S$ (so $V_S=1/S$, $V_{SS}=-1/S^2$, $V_t=0$) under GBM. What is the drift of $d(\ln S)$, and why is it $\mu-\tfrac12\sigma^2$?

7.5 (Monte Carlo.) A Monte Carlo price uses 2,500 paths and you want to cut the error to one-fifth. How many paths do you need?

7.6 (Connection.) In one sentence, what is the $\tfrac12\sigma^2 S^2 V_{SS}$ term in Itô's lemma, in the language of Chapter 2?

Solutions

Symbols & terms in this chapter

Next: Chapter 8 — Black–Scholes & the Greeks, where Itô's lemma becomes a pricing formula.

Chapter 8 — Black–Scholes & the Greeks

Maps to: the per-option sensitivities (Δ, Γ, Θ, ν) that Chapter 9 aggregates into the gamma flip and walls. Lab: implement Black–Scholes, validate it against Monte Carlo, and watch gamma sharpen into a wall as expiry nears.

Learning objectives

By the end you can: (1) state the delta-hedging argument and the Black–Scholes PDE; (2) write the BS formula and read $N(d_1)$, $N(d_2)$; (3) give the closed forms and meaning of delta, gamma, theta, vega; (4) explain the theta–gamma relationship; (5) show where gamma peaks — at the strike, blowing up as $T\to0$ — the source of sharp 0-DTE walls; (6) compute it all in the lab.

8.1 Motivation: to find the walls, price the option

Chapter 9's gamma flip and walls are aggregates of every option's gamma. To get gamma you need a pricing model, and Black–Scholes is that model — built directly on Chapter 7's Itô's lemma plus one clean idea: a continuously delta-hedged option carries no risk, so it must earn the risk-free rate. From that argument fall the formula and the Greeks, and — most important for trading the open — the fact that gamma is enormous and razor-sharp at the strike of a near-expiry option. That sharpness is the wall.

8.2 The Black–Scholes PDE — the hedging argument

Hold an option $V(S,t)$ and short $\Delta=\partial V/\partial S$ of the underlying. By Itô's lemma (Chapter 7), the random $dW$ terms of the option and the hedge cancel — the combined portfolio is instantaneously riskless. A riskless portfolio can only earn the risk-free rate $r$ (else arbitrage). Setting the portfolio's Itô drift equal to $r$ times its value gives the Black–Scholes PDE: $$\boxed{\ \frac{\partial V}{\partial t} + rS\frac{\partial V}{\partial S} + \tfrac12\sigma^2 S^2\frac{\partial^2 V}{\partial S^2} = rV\ }$$ Notice what happened: the drift $\mu$ vanished. Hedging removed the directional bet, so the option's value depends on volatility and rates but not on which way you think the market is going — the risk-neutral principle. And there again is the $\tfrac12\sigma^2 S^2\,\partial^2 V/\partial S^2$ gamma term, now driving the price itself.

8.3 The Black–Scholes formula

Solving the PDE for a European call gives the famous formula: $$\boxed{\ C = S\,N(d_1) - K e^{-rT} N(d_2)\ },\qquad d_1 = \frac{\ln(S/K) + (r+\tfrac12\sigma^2)T}{\sigma\sqrt{T}},\quad d_2 = d_1 - \sigma\sqrt{T}.$$ $N(\cdot)$ is the standard Normal CDF (Chapter 1's $\Phi$). Read it: $N(d_1)$ is the option's delta, and $N(d_2)$ is the risk-neutral probability the call finishes in the money. The whole price is "expected payoff if exercised, discounted" — with the two $N$'s doing the probability bookkeeping.

Worked example 1 ($d_1$, $d_2$, and price). For $S=K=100$, $T=1$, $r=0$, $\sigma=0.2$: $d_1=\frac{0+(0+0.02)(1)}{0.2}=0.1$, $d_2=0.1-0.2=-0.1$. Then $C=100\,N(0.1)-100\,N(-0.1)=100(0.5398-0.4602)=7.97$. An at-the-money one-year call on a \$100 stock at 20% vol is worth about \$7.97.

8.4 The Greeks — closed forms and meaning

Differentiate the formula (Chapter 2) to get the sensitivities:

Here $\phi$ is the standard Normal density. The BS PDE ties two of them together: a delta-hedged position's value change is $$d\Pi \approx \tfrac12\Gamma\,(dS)^2 + \Theta\,dt.$$ Long gamma ($\Gamma>0$) means you gain from movement ($\tfrac12\Gamma(dS)^2>0$) but pay for it through negative theta — you rent convexity by the day. That is the same $\tfrac12\Gamma\,dS^2$ from Chapter 2, now with its rental price attached.

Worked example 2 (delta as a hedge ratio). With the parameters above, $\Delta=N(0.1)=0.54$. To hedge one long call you short 0.54 shares; as the stock rises the call's delta climbs toward 1 and you short more — buying low, selling high in reverse, the dealer's stabilizing dance from Chapter 2.

8.5 Where gamma lives — and why walls are sharp

Gamma is $\phi(d_1)/(S\sigma\sqrt{T})$. Two facts make it the key to the open:

• It peaks at the strike (where $d_1\approx0$, so $\phi(d_1)$ is largest) and falls off away from it — gamma is local to the strike.
• It scales like $1/\sqrt{T}$, so as expiry approaches it blows up. A 0-DTE option has enormous gamma in a razor-thin band around its strike.

Put those together and you get a sharp wall: near expiry, dealer gamma is concentrated at the big strikes and towering. Price pins to those strikes because the hedging flows there are violent and the gamma is huge. This is exactly the structure Chapter 9 aggregates across the chain into the flip and the call/put walls.

Interactive (rotate / zoom): the gamma surface over spot and time-to-expiry — a ridge running along the strike that rises and sharpens as $T\to0$. That ridge is the wall the terminal draws.

Worked example 3 (gamma explodes into expiry). ATM gamma scales as $1/\sqrt{T}$. Going from 30 days to 1 day to expiry multiplies gamma by $\sqrt{30/1}\approx5.5$. Same strike, same stock — 5.5× the gamma and a far narrower band. That's why 0-DTE walls are both taller and sharper than monthly ones.

8.6 Put–call parity

Calls and puts on the same strike are locked together by put–call parity: $$C - P = S - K e^{-rT}.$$ Differentiate twice in $S$ and the $S$ and $K$ terms vanish — so calls and puts at a strike have identical gamma. That's why Chapter 9 can aggregate gamma by strike without fussing over call-vs-put: at each strike, both contribute the same curvature.

8.7 Chapter summary

• Delta-hedging makes an option riskless instantaneously → the BS PDE; the drift $\mu$ drops out (risk-neutral).
• BS call: $C=S\,N(d_1)-Ke^{-rT}N(d_2)$; $N(d_1)=$ delta, $N(d_2)=$ risk-neutral ITM probability.
• Greeks: $\Delta=N(d_1)$, $\Gamma=\phi(d_1)/(S\sigma\sqrt{T})$, $\nu=S\phi(d_1)\sqrt{T}$; a hedged position earns $\tfrac12\Gamma(dS)^2+\Theta\,dt$ — gamma vs theta.
• Gamma peaks at the strike and grows like $1/\sqrt{T}$ → sharp, tall 0-DTE walls.
• Put–call parity → calls and puts share gamma at a strike.

Key formulas to carry forward:

Exercises

8.1 ($d_1$, $d_2$.) For $S=105$, $K=100$, $T=0.25$, $r=0$, $\sigma=0.2$, compute $d_1$ and $d_2$.

8.2 (Delta.) Using $d_1$ from 8.1, find the call's delta ($N(d_1)$; $N(0.6)\approx0.726$). How many shares hedge one call?

8.3 (Gamma & expiry.) ATM gamma is $\propto 1/\sqrt{T}$. By what factor does it change going from 16 days to 1 day to expiry?

8.4 (Theta–gamma.) A delta-hedged long gamma book profits from a large move but bleeds on a quiet day. Explain both using $d\Pi \approx \tfrac12\Gamma(dS)^2+\Theta\,dt$.

8.5 (Risk-neutral.) Why does the drift $\mu$ not appear in the Black–Scholes price?

8.6 (Parity.) A call is worth \$7, the stock is \$100, the strike \$100, $r=0$. What is the put worth?

Solutions

Symbols & terms in this chapter

Next: Chapter 9 — Dealer Gamma Exposure (GEX): Building the Cage, where these Greeks aggregate into the flip and walls.

Chapter 9 — Dealer Gamma Exposure (GEX): Building the Cage

Maps to: the gamma flip, call wall, and put wall the terminal computes (lib/uw.ts getNqLevels, computeGamma). Lab: build a synthetic option chain, aggregate net GEX, and locate the flip.

Learning objectives

By the end you can: (1) explain dealer positioning and why the sign of dealer gamma dampens or amplifies; (2) aggregate per-strike gamma into net GEX(S); (3) locate the gamma flip as its zero-crossing; (4) read the call and put walls off the gamma profile; (5) connect long/short gamma to the pinning vs trending regimes; (6) name the honest caveats (why the terminal reports a normalized γ tilt, not a fake dollar GEX).

9.1 Motivation: the cage, finally

Nine chapters have converged on this. The gamma cage the terminal draws — a flip in the middle, a call wall above, a put wall below — is not an indicator someone invented. It is Chapter 8's gamma, summed across every strike on the chain, weighted by how much is open, and interpreted through the dealer hedging you already understand from Chapter 2. This chapter assembles those pieces into the levels you trade around every open.

9.2 Dealers, positioning, and the sign of gamma

Market makers take the other side of what customers trade, then delta-hedge (Chapter 8). What matters for the tape is the sign of the dealers' net gamma at a given price:

• Dealers net long gamma → to stay hedged they sell rallies and buy dips — supplying liquidity, dampening moves, pulling price toward a pin.
• Dealers net short gamma → they buy rallies and sell dips — chasing the move, amplifying it.

That's the exact $\tfrac12\Gamma\,dS^2$ mechanism of Chapter 2, now read at the level of the whole dealer book. The typical picture: dealers are long gamma above a certain price and short gamma below it, because of how call and put open interest is distributed.

9.3 Net GEX — aggregating the chain

Each strike contributes gamma exposure proportional to its gamma (Chapter 8), the open interest (OI) sitting there, the contract multiplier, and a scale factor of spot squared. Summing across strikes with the dealer sign gives net gamma exposure as a function of spot: $$\boxed{\ \text{GEX}(S) = \sum_{k}\ \text{sign}_k\cdot \Gamma(S,K_k,T)\cdot \text{OI}_k\cdot m\cdot S^2\ }$$ GEX(S) tells you, at each hypothetical price, whether the aggregate dealer book is long or short gamma — and therefore whether the market is being damped or amplified right there.

9.4 The gamma flip

The gamma flip is simply the spot where net GEX crosses zero: $$\boxed{\ \text{GEX}(S^\*) = 0\ }$$ It's the boundary between regimes. Above the flip, dealers are net long gamma and the tape is sticky — fades work, extremes get pulled back toward the pin. Below the flip, dealers flip net short and the tape accelerates — breaks run, momentum feeds itself. That single level reframes the whole session: the same news is mean-reverting above the flip and trend-igniting below it. Finding $S^\*$ is a root-find on GEX(S).

Interactive (rotate / zoom): net GEX over spot and time-to-expiry. Watch the zero-crossing (the flip) firm up and the profile steepen as expiry approaches — the cage tightening.

Worked example 1 (a per-strike contribution). One strike with dealer-long gamma $\Gamma=0.02$, OI $=5{,}000$ contracts, multiplier $m=100$, spot $S=100$: its GEX contribution is $0.02\cdot5000\cdot100\cdot100^2 = 1.0\times10^{8}$ in raw units. Sum thousands of these across strikes — some positive, some negative — and the net is GEX(S).

9.5 The walls

The call wall and put wall are the strikes carrying the largest gamma concentration on each side of spot:

• Call wall — the big positive-gamma strike above price. Dealers there are heavily long gamma, so rallies into it get sold; it acts as an upside cap / magnet.
• Put wall — the big strike below price, a support shelf and downside magnet.

They are the peaks of the per-strike gamma × OI profile. Because gamma peaks at each strike and towers near expiry (Chapter 8), on a 0-DTE session the walls are tall and sharp, and price genuinely pins to them.

9.6 Why price pins — the two regimes

Put the pieces together and you get the two behaviors the desk read keys on:

• Long-gamma (above the flip, near the walls): heavy dealer selling into strength and buying into weakness pins price and compresses the range. Fade the edges; expect mean reversion toward the pin.
• Short-gamma (below the flip): dealer hedging chases the move, so a break expands and trends. Trade the break, not the fade.

Worked example 2 (reading the regime). Net GEX at spot is positive and price sits just under the call wall. Regime: long gamma → dealers sell rallies. Playbook: fade a tag of the wall back toward the pin, size for a compressed range. Flip the sign — negative GEX below the flip — and the same wall becomes a breakout trigger, not a fade.

9.7 Honest caveats

GEX is a model, and its inputs are proxies. Two cautions the terminal takes seriously:

• Dealer sign is assumed, and OI is a stock (not the flow) — so the level of GEX is uncertain even when the shape is informative.
• Units don't cleanly convert to dollars. Different feeds normalize gamma differently, so a headline "net GEX = \$X billion" is often apples-to-oranges. That's why the terminal reports a normalized γ tilt in $[-1,+1]$ (net ÷ gross gamma) and a regime label (long/short), which are robust, rather than a fake dollar figure. The flip and walls — relative structure — are the trustworthy outputs.

9.8 Chapter summary

• Dealers hedge; the sign of their net gamma decides dampening vs amplifying (Chapter 2 at book scale).
• Net GEX(S) = per-strike gamma × OI × $m$ × $S^2$, summed with the dealer sign.
• The gamma flip is where GEX(S) = 0 — long-gamma (sticky) above, short-gamma (trendy) below.
• Call/put walls are the largest gamma-OI strikes above/below spot — the cap and the shelf; sharp near expiry.
• GEX is a model: trust the shape (flip, walls, regime), not a dollar level — hence the normalized γ tilt.

Key formulas to carry forward:

Exercises

9.1 (Contribution.) A strike has dealer gamma $\Gamma=0.015$, OI $=8{,}000$, $m=100$, $S=50$. Compute its GEX contribution (raw units).

9.2 (Flip.) Net GEX is $+2.0$ at $S=101$ and $-1.5$ at $S=99$ (arbitrary units). Roughly where is the gamma flip, and what changes as price crosses it?

9.3 (Walls.) Above spot, strike 110 has gamma×OI $=900$ and strike 115 has $400$; below spot, strike 90 has $1{,}200$ and strike 95 has $500$. Name the call wall and the put wall.

9.4 (Regime.) Dealers are net short gamma and price breaks below the put wall. Which way do dealers trade to re-hedge, and what does it do to the move?

9.5 (Mechanism.) In one sentence, why does a long-gamma dealer's hedging pull price toward a pin?

9.6 (Caveat.) Why does the terminal report a normalized γ tilt instead of a dollar GEX number?

Solutions

Symbols & terms in this chapter

Next: Chapter 10 — Logistic Regression from Scratch, where the openBias probability is built — Part IV begins.

Chapter 10 — Logistic Regression from Scratch

Maps to: the openBias probability P(up 5m) (lib/openBias.ts) — the directional tilt and the coin flip. Lab: fit a logistic model by gradient descent and check its calibration.

Learning objectives

By the end you can: (1) say why linear regression can't model a probability; (2) use odds, log-odds, and the sigmoid; (3) write the logistic model $p=\text{sigmoid}(\beta\cdot x)$; (4) fit it by maximum likelihood / gradient descent and read the beautiful gradient $\sum(p_i-y_i)x_i$; (5) interpret coefficients as log-odds effects and see the coin flip at $z=0$; (6) check calibration.

10.1 Motivation: predicting a probability

The openBias model turns a handful of factors — the overnight volatility shift, rates, the dollar — into a single number: $P(\text{up in 5 min})$. That is a classification problem (up or down) reported as a probability, and the standard tool is logistic regression. This chapter builds it from the ground up, and along the way makes precise the coin-flip open you've met since Chapter 1: it is simply the point where the model outputs $0.5$.

10.2 Why not just use a line?

Chapter 4's linear regression predicts any real number — but a probability must live in $[0,1]$. Fit a line to up/down outcomes and it will happily predict $1.3$ or $-0.2$, which is meaningless. We need a model whose output is always a valid probability. The fix is to squash the line's output through a function that maps the whole real line into $(0,1)$ — the sigmoid.

10.3 Odds, log-odds, and the sigmoid

Start from odds, the ratio of up to down: $\text{odds}=\dfrac{p}{1-p}$. Odds run from $0$ to $\infty$. Take their logarithm and you get the log-odds (the logit), which runs over the whole real line: $$\text{logit}(p) = \ln\frac{p}{1-p}.$$ Now the trick: model the log-odds as a linear function of the factors, $\text{logit}(p)=\beta\cdot x=\beta_0+\beta_1 x_1+\cdots$. Inverting gives the probability through the sigmoid: $$\boxed{\ p = \frac{1}{1+e^{-z}},\qquad z = \beta\cdot x\ }$$ The sigmoid is an S-curve: it sends $z=0$ to $p=0.5$, large positive $z$ toward $1$, large negative toward $0$ — always a valid probability.

Worked example 1 (odds and log-odds). $p=0.6$ gives odds $=0.6/0.4=1.5$ and log-odds $=\ln 1.5=0.405$. Going back, $\text{sigmoid}(0.405)=1/(1+e^{-0.405})=0.6$ — the logit and sigmoid are inverses.

10.4 Fitting by maximum likelihood

Unlike OLS, there's no tidy formula. We choose the coefficients that make the observed outcomes most likely. Each session is a Bernoulli trial: probability $p_i$ of the up-outcome it actually had. Maximizing the likelihood is the same as minimizing the log-loss (cross-entropy): $$\boxed{\ \mathcal{L}(\beta) = -\sum_i\Big[y_i\ln p_i + (1-y_i)\ln(1-p_i)\Big]\ }$$ where $y_i\in\{0,1\}$ is the outcome and $p_i=\text{sigmoid}(\beta\cdot x_i)$. Log-loss punishes confident wrong predictions harshly — a model that says 99% and is wrong pays a huge penalty. There's no closed form for the minimizer, so we descend to it.

10.5 Gradient descent — and a beautiful gradient

To minimize the loss, step downhill: repeatedly nudge $\beta$ against the gradient. The gradient of the logistic log-loss has a strikingly clean form — the residual times the feature: $$\boxed{\ \frac{\partial \mathcal{L}}{\partial \beta} = \sum_i (p_i - y_i)\,x_i\ }$$ Read it: where the prediction $p_i$ overshoots the outcome $y_i$, push the coefficients down in the direction of that feature. Update $\beta \leftarrow \beta - \eta\,\frac{\partial\mathcal L}{\partial\beta}$ with a small learning rate $\eta$, and repeat until it settles. One practical must: standardize the factors (subtract the mean, divide by the std, Chapter 3) so they're on comparable scales — otherwise the descent zig-zags and the coefficients aren't comparable.

Worked example 2 (one gradient step). Three points $x=(0.5,-1,2)$, outcomes $y=(1,0,1)$, starting at $\beta=0$ so every $p_i=0.5$. The intercept gradient is $\sum(p_i-y_i)=(0.5-1)+(0.5-0)+(0.5-1)=-0.5$; the slope gradient is $\sum(p_i-y_i)x_i=(-0.5)(0.5)+(0.5)(-1)+(-0.5)(2)=-1.75$. Step against these and both coefficients rise — the fit begins to lean the right way.

10.6 Reading the model — and the coin flip

Each coefficient is a log-odds effect: a one-unit (standardized) increase in factor $x_j$ shifts the log-odds by $\beta_j$, i.e. multiplies the odds by $e^{\beta_j}$ (the odds ratio). Stack the factors and the total $z=\beta\cdot x$ lands you somewhere on the S-curve. Two consequences you already know by name:

• When the factors are neutral or cancel out, $z\approx0$ and $p\approx0.5$ — the coin-flip open. It's not a special case bolted on; it's the center of the sigmoid.
• When the factors agree, they push $z$ well away from zero and $p$ toward a confident tilt — the directional read.

Interactive (rotate / zoom): the probability surface $p(x_1,x_2)=\text{sigmoid}(\beta_0+\beta_1 x_1+\beta_2 x_2)$ over two factors — an S-shaped sheet whose $p=0.5$ contour is the decision boundary (the coin-flip line).

Worked example 3 (the coin flip, exactly). Suppose two standardized factors have coefficients $+1.2$ and $-1.2$ and today they read $x_1=+1$, $x_2=+1$. Then $z=\beta_0+1.2-1.2=\beta_0$; if the intercept is $0$, $z=0$ and $p=0.5$. The factors disagreed and canceled — the model returns a coin flip straight from the arithmetic, exactly the openBias "factors mixed → no directional edge."

10.7 Calibration — does 60% mean 60%?

A probability model is only useful if its numbers are honest. Calibration asks: of all the times the model said "60%," did the up-outcome really happen about 60% of the time? To check, bin the predictions and compare each bin's average prediction to the observed frequency; a well-calibrated model tracks the diagonal. Two summary metrics: log-loss (lower is better, punishes overconfidence) and AUC (ranking quality). A model can discriminate well yet be miscalibrated — which is why openBias's probabilities are worth checking against realized outcomes, not just trusted.

10.8 Chapter summary

• A probability needs $(0,1)$ output, so model the log-odds as linear and squash with the sigmoid.
• Logistic model: $p=\text{sigmoid}(\beta\cdot x)$; fit by maximum likelihood = minimizing log-loss.
• Gradient $=\sum(p_i-y_i)x_i$ (residual × feature); descend with a small step; standardize factors.
• Coefficients are log-odds effects ($e^\beta$ = odds ratio); $z=0\Rightarrow p=0.5$ is the coin flip.
• Always check calibration — that a 60% really means 60%.

Key formulas to carry forward:

Exercises

10.1 (Log-odds.) Convert $p=0.75$ to odds and log-odds. Convert log-odds $z=-1$ back to a probability.

10.2 (Odds ratio.) A standardized factor has coefficient $\beta=0.7$. By what factor does a one-unit increase multiply the odds of an up-open?

10.3 (Coin flip.) Factors give $z=\beta\cdot x = 0$. What is $p$, and what does that mean for the read?

10.4 (Gradient sign.) The model predicts $p=0.8$ for a session that closed down ($y=0$) with feature $x=+1$. What is that point's contribution to the slope gradient $\sum(p-y)x$, and which way will the coefficient move?

10.5 (Log-loss.) Two models both predict the up-outcome correctly, but one says $p=0.6$ and the other $p=0.95$. Which has the lower log-loss on that point, and why?

10.6 (Calibration.) The model's "70%" bucket resolves up only 55% of the time. Is it well-calibrated? What's the fix in spirit?

Solutions

Symbols & terms in this chapter

Next: Chapter 11 — Multivariate Factors & Multicollinearity, why "more signals" isn't more edge.

Chapter 11 — Multivariate Factors & Multicollinearity

Maps to: the openBias factor set (lib/openBias.ts, lib/snapshot.ts) — and why econ-surprise was left out. Lab: watch collinear predictors make coefficients explode, and measure redundancy with VIF.

Learning objectives

By the end you can: (1) compute covariance and correlation and read a covariance matrix; (2) explain multicollinearity and why it destabilizes coefficients; (3) measure redundancy with the Variance Inflation Factor; (4) see why a factor spanned by others adds no signal — the econ-surprise decision; (5) get the idea of PCA as decorrelating factors; (6) decide whether a new factor is worth adding.

11.1 Motivation: why not just add more factors?

openBias combines a few factors — the overnight volatility shift, rates, the dollar. It's tempting to throw in more: econ-surprise, sentiment, put/call ratios. But adding factors that move together with the ones you have doesn't add information — it adds instability. This chapter is the mathematics of "more signals ≠ more edge," and it explains a real decision: econ-surprise was tested and left out of openBias because it was already explained by rates and the dollar.

11.2 Covariance and correlation

Covariance measures how two variables move together: $$\text{Cov}(X,Y) = \mathbb{E}\big[(X-\mu_X)(Y-\mu_Y)\big].$$ It's positive when they rise together, negative when one rises as the other falls — but its size depends on the units. Correlation normalizes it to a unitless number in $[-1,1]$: $$\boxed{\ \rho = \frac{\text{Cov}(X,Y)}{\sigma_X\,\sigma_Y}\ }$$ $\rho=\pm1$ is perfect linear co-movement, $\rho=0$ is no linear relationship. It's the same dot-product idea from Chapter 4: correlation is the cosine of the angle between the two centered data vectors.

11.3 The covariance matrix

With many factors, all the pairwise covariances stack into the covariance matrix $\Sigma$ (its normalized version is the correlation matrix). The diagonal holds each factor's variance; the off-diagonals its co-movements. That single object encodes the entire linear dependence structure of your factor set — and reading it is the first thing you do before trusting a multi-factor model.

Worked example 1 (covariance → correlation). Two factors have $\text{Cov}(X,Y)=0.36$, $\sigma_X=0.6$, $\sigma_Y=0.8$. Then $\rho=0.36/(0.6\cdot0.8)=0.36/0.48=0.75$ — strongly correlated. If instead $\text{Cov}=0.05$, $\rho=0.10$ — nearly independent.

11.4 Multicollinearity — when the model can't tell factors apart

Recall the least-squares solution $\hat\beta=(X^\top X)^{-1}X^\top y$ (Chapter 4). When two columns of $X$ are highly correlated, $X^\top X$ becomes nearly singular and its inverse blows up. Concretely, the model can't decide how to split the shared effect between the two factors — it can put a big positive weight on one and a big negative on the other and get almost the same fit. The result: wildly unstable, high-variance coefficients that flip sign from sample to sample, even though the prediction is fine.

Interactive (rotate / zoom): the least-squares loss over two coefficients when the predictors are collinear — a long, flat ravine instead of a tight bowl. Every point along the ravine floor fits almost equally well, so the minimum is barely determined. That flatness is coefficient instability.

Worked example 2 (the trade-off). If $x_2\approx x_1$ and the true effect is $y\approx x_1$, then $\beta_1 x_1+\beta_2 x_2 \approx (\beta_1+\beta_2)x_1$. The fit only pins down $\beta_1+\beta_2\approx1$ — so $(\beta_1,\beta_2)=(1,0)$, $(0,1)$, or $(5,-4)$ all fit about the same. The pair is unidentifiable; only the sum is real.

11.5 The Variance Inflation Factor

Quantify a factor's redundancy by regressing it on all the other factors and taking the $R^2$ of that fit. The Variance Inflation Factor is $$\boxed{\ \text{VIF}_j = \frac{1}{1-R_j^2}\ }$$ If a factor is well explained by the others ($R_j^2$ near 1), its VIF is huge — its coefficient's variance is inflated many-fold. Rules of thumb: VIF around 1 is clean, above ~5 is worrying, above ~10 means the factor is essentially a linear combination of the rest — drop it or combine it.

Worked example 3 (VIF). A candidate factor regressed on the existing ones gives $R^2=0.90$. Then $\text{VIF}=1/(1-0.90)=10$ — its coefficient variance is 10× inflated. The factor adds almost no independent information; keeping it just destabilizes the model.

11.6 Why econ-surprise stayed out of openBias

This is the decision made concrete. Econ-surprise was a plausible directional factor — but at the 9:30 open it is largely spanned by rates and the dollar: a hot inflation print moves rates and the dollar, and those are already in the model. Regressing surprise on rates + dollar returns a high $R^2$ (high VIF), so surprise adds collinearity, not signal. The backtest confirmed it didn't earn its place at that horizon. The lesson is general: a factor is only worth adding if it carries information orthogonal to what you already have — new direction, not a louder echo.

Worked example 4 (redundant by construction). If $\text{surprise}=0.9\,\text{rates}-0.4\,\text{dollar}+\varepsilon$ with small $\varepsilon$, then regressing surprise on rates and the dollar yields $R^2$ near 0.9, VIF near 10. Mathematically it is the other two factors in disguise — including it can only hurt.

11.7 A first taste of PCA

When factors are correlated, you can rotate them into a new set of uncorrelated axes — the principal components — ordered so the first captures the most variance, the second the next, and so on. The components are the eigenvectors of the covariance matrix; keeping the first few is dimensionality reduction that decorrelates your factors and discards the redundant directions. In practice you might replace three correlated macro factors with one or two orthogonal components, sidestepping multicollinearity entirely.

11.8 Chapter summary

• Covariance measures co-movement; correlation $\rho$ normalizes it to $[-1,1]$ (a cosine of angle).
• The covariance matrix encodes the whole linear dependence structure — read it first.
• Multicollinearity makes $(X^\top X)^{-1}$ blow up → unstable, sign-flipping coefficients; only their sum is pinned.
• VIF $=1/(1-R_j^2)$ measures redundancy; high VIF ⇒ drop or combine.
• Econ-surprise was left out because rates + dollar already span it — add a factor only if it's orthogonal.
• PCA rotates correlated factors into uncorrelated components — decorrelation by design.

Key formulas to carry forward:

Exercises

11.1 (Correlation.) $\text{Cov}(X,Y)=-0.24$, $\sigma_X=0.4$, $\sigma_Y=1.0$. Compute $\rho$. Are they positively or negatively related?

11.2 (VIF.) A factor regressed on the others has $R^2=0.80$. Compute its VIF. Keep it or drop it?

11.3 (Collinearity.) Two predictors are nearly identical and the fit pins $\beta_1+\beta_2\approx2$. Give three different $(\beta_1,\beta_2)$ pairs that fit about equally well. What does this say about interpreting either coefficient alone?

11.4 (Orthogonality.) When is a new factor worth adding to openBias, in terms of its relationship to the existing factors?

11.5 (PCA.) Three macro factors are strongly correlated. In words, what does replacing them with their first principal component achieve?

11.6 (Econ-surprise.) In one sentence, why did econ-surprise not earn a slot in openBias despite being a sensible directional signal?

Solutions

Symbols & terms in this chapter

Next: Chapter 12 — Backtest Statistics & the Ways They Deceive, the discipline of an honest edge. Part V begins.

Chapter 12 — Backtest Statistics & the Ways They Deceive

Maps to: the backtest scripts (scripts/backtest-*.mjs) and the accuracy tracker (lib/scoreBrief.ts). Lab: manufacture a "winning" strategy out of pure noise, then watch it die out-of-sample.

Learning objectives

By the end you can: (1) treat a backtest as a hypothesis test and put a standard error on a Sharpe ratio; (2) explain why in-sample results are optimistic and how much; (3) quantify the multiple-testing trap and correct for it (Bonferroni, expected max-t, the deflated Sharpe); (4) name the classic biases — look-ahead, survivorship, data-snooping; (5) compute a minimum track-record length; (6) fold in transaction costs. In short: know when an edge is real.

12.1 Motivation: a backtest is a hypothesis test you run on yourself

Every backtest asks one question — does this strategy have an edge, or did I just get lucky? That's exactly the $H_0$ vs $H_1$ machinery from Chapter 3, now pointed at your own P&L. The danger is that you are both the experimenter and the thing being fooled: you try ideas until one "works," then believe it. This chapter is the set of guardrails that keep a backtest honest — the same discipline behind refusing to "fix" the range model on six good days ([[fp-terminal-daily-email]], Chapter 3).

12.2 The Sharpe ratio and its standard error

The Sharpe ratio measures return per unit of risk: $$\text{SR} = \frac{\bar r}{s_r}\quad(\text{per period, annualized by }\times\sqrt{\text{periods/yr}}).$$ It is an estimate from a finite sample, so it carries a standard error. For returns that are roughly independent, $$\boxed{\ \text{SE}(\text{SR}) \approx \sqrt{\frac{1+\tfrac12\text{SR}^2}{n}}\ }$$ and the t-statistic for "is the true Sharpe $>0$?" is essentially $$t \approx \text{SR}\sqrt{n}.$$ So a Sharpe is only as trustworthy as the number of observations behind it. A dazzling SR on 20 trades is a rounding error away from zero.

Worked example 1 (Sharpe t-stat). A strategy shows a daily Sharpe of $0.15$ over $n=252$ days. Then $t\approx0.15\sqrt{252}\approx2.38$ — significant at 5%. But the same daily Sharpe over $n=30$ days gives $t\approx0.15\sqrt{30}\approx0.82$ — indistinguishable from luck. Identical performance, opposite verdict, because of sample size.

12.3 In-sample optimism: why the backtest always looks better

When you fit and evaluate on the same data, the strategy gets to memorize the noise — the in-sample Sharpe is biased upward. Out-of-sample, that borrowed luck evaporates. The more knobs you tuned (parameters, thresholds, which factors to include), the bigger the gap. This is the Chapter 6 bias–variance story wearing a P&L costume: complexity buys in-sample fit and pays for it out-of-sample.

12.4 The multiple-testing trap

Here's the quiet killer. Test one worthless strategy and there's a 5% chance it clears $t>1.96$ by luck. Test $N$ of them and the chance that at least one looks significant is $1-(1-0.05)^N$ — for $N=20$ that's about 64%. Search hard enough over pure noise and you are guaranteed a "winner."

Two defenses: - Bonferroni: to keep the overall false-positive rate at $\alpha$ across $N$ trials, require each to beat $\alpha/N$ (e.g. $0.05/20=0.0025$ → $|t|>3.0$, not $1.96$). - Expected maximum t: under the null, the largest of $N$ independent t-stats grows like $\mathbb{E}[\max]\approx\sqrt{2\ln N}$. Your best backtest has to beat that bar, not the single-test bar.

Interactive (rotate / zoom): the Sharpe you'd need just to beat randomness at 95%, as a function of how many strategies you tried and how much data each had. More trials raises the bar; more data lowers it. This surface is the deflated Sharpe made visible.

Worked example 2 (best of many). You backtest $N=50$ variations, all truly worthless. The expected largest t-stat is $\approx\sqrt{2\ln 50}\approx2.8$, and the maximum often lands above 3. So a $t=2.5$ "winner" from a 50-way search is below what noise alone produces — no edge at all, despite clearing the naive 1.96.

12.5 The classic biases

Even one honest test can be corrupted before it starts:

• Look-ahead bias — using information that wasn't available yet (tomorrow's close, a revised figure). The scoreBrief.ts tracker only ever grades a call against what came after it, never before.
• Survivorship bias — testing on the names that lived; the delisted losers are invisible and flatter you.
• Data-snooping / p-hacking — trying cutoffs, windows, and universes until something passes, then reporting only the winner. It's multiple testing you forgot to count.
• Overfitting to the backtest — tuning until the equity curve looks good is fitting the noise directly.

12.6 The deflated Sharpe and minimum track record

Put §12.2 and §12.4 together. Because searching inflates the bar, the honest question is "is my Sharpe above what the best of $N$ trials would give under the null?" — the deflated Sharpe ratio. Inverting the t-stat gives a minimum track-record length: the number of observations needed for a claimed Sharpe to clear significance, $$n^\* \approx \Big(\frac{t_\alpha}{\text{SR}}\Big)^2\big(1+\tfrac12\text{SR}^2\big).$$ Small Sharpe ⇒ you need a lot of history before believing it.

Worked example 3 (minimum track record). To confirm a daily Sharpe of $\text{SR}=0.10$ at $t=2$: $n^\*\approx(2/0.10)^2\cdot(1+0.005)\approx 400\cdot1.005\approx402$ days — about 1.6 years of daily data. Anything shorter can't distinguish this edge from zero.

Worked example 4 (the +71% on 6 days, revisited). Six winning days is $n=6$; even a huge raw return gives $t\approx\text{SR}\sqrt6$, and the 95% CI on the win-rate spans almost the whole $[0,1]$ interval (Chapter 3's "$-3\%$ to $+145\%$"). Statistically it is silence — which is why the range model was not re-fit on it.

12.7 Costs turn paper gold into lead

A backtest without frictions is fiction. Each round trip pays the spread, commissions, and slippage; subtract them from every trade. A strategy that trades often lives or dies on cost: a per-trade edge of a few ticks can vanish entirely once the spread is charged.

Worked example 5 (cost drag). A signal earns $+2.0$ NQ points per trade gross and fires 8×/day. Round-trip cost (spread + slippage + commission) is $\approx1.4$ points. Net $=0.6$ pt/trade $\to 4.8$ pt/day gross of the gross-looking $16$ — a two-thirds haircut. Many "edges" are entirely inside the spread.

12.8 Doing it right: out-of-sample by construction

The cure is to never let the strategy see its exam. Walk-forward (fit on a rolling past window, trade the next block, repeat), a strict hold-out set touched once, and purged/embargoed cross-validation (drop samples around each test block so overlapping labels don't leak) all estimate performance on data the model never tuned to — exactly the train/validate/test discipline of Chapter 6, applied to money.

12.9 Chapter summary

• A backtest is a hypothesis test; a Sharpe needs a standard error: $t\approx\text{SR}\sqrt n$.
• In-sample results are optimistic; the gap grows with the number of knobs you turned.
• Multiple testing manufactures winners — correct with Bonferroni ($\alpha/N$) or the expected-max-t bar $\sqrt{2\ln N}$; report the deflated Sharpe.
• Beware look-ahead, survivorship, snooping, overfitting — quiet ways to lie before the test even runs.
• Demand a minimum track record; a small Sharpe needs a lot of history.
• Subtract costs on every trade — many edges live entirely inside the spread.
• Validate out-of-sample by construction: walk-forward, hold-out, purged CV.

Key formulas to carry forward:

Exercises

12.1 (Sharpe t-stat.) A daily Sharpe of $0.12$ over $n=180$ days. Compute $t$. Significant at 5% ($t>1.96$)?

12.2 (Family-wise error.) You test 15 independent worthless strategies at $\alpha=0.05$. What's the chance at least one looks significant? What Bonferroni threshold keeps the overall rate at 5%?

12.3 (Best of many.) You searched $N=100$ strategies over noise. Estimate the expected largest t-stat. Does a reported $t=2.6$ impress you?

12.4 (Minimum track record.) How many daily observations to confirm a Sharpe of $0.08$ at $t=2$?

12.5 (Costs.) A signal earns $1.5$ points/trade gross, trades 10×/day, round-trip cost $1.2$ points. Net points per day? What fraction of the gross survives?

12.6 (Bias spotting.) Name the bias: (a) you backtest on today's S&P 500 members over the last 20 years; (b) your entry uses the day's closing price to decide a trade made at the open.

Solutions

Symbols & terms in this chapter

Next: Chapter 13 — Position Sizing & the Kelly Criterion, turning an edge into the right bet size.

Chapter 13 — Position Sizing & the Kelly Criterion

Maps to: the conviction → risk step the Desk Read hands you (lib/scoreBrief.ts conviction) — the terminal gives the signal; sizing is yours. Lab: simulate wealth under different bet fractions and watch Kelly maximize growth while overbetting ruins.

Learning objectives

By the end you can: (1) explain why long-run wealth is a geometric (multiplicative) process and why you maximize expected log wealth; (2) derive and apply the Kelly fraction for a discrete bet and for a continuous strategy; (3) connect optimal growth to the Sharpe ratio ($g^\*=\text{SR}^2/2$); (4) explain why full Kelly is too aggressive and use fractional Kelly; (5) see how estimation error (Chapter 12) forces you to bet less; (6) reason about drawdown and ruin.

13.1 Motivation: a real edge is only half the problem

Chapter 12 was about whether you have an edge. This one is about how much to bet on it. Size too small and you leave growth on the table; size too big and a normal losing streak wipes you out — even with a genuine edge. The terminal delivers a direction and a conviction; converting that into a position size is the trader's decision, and it has a mathematically optimal answer.

13.2 Why maximize expected log wealth

Trading is multiplicative: a $-50\%$ then $+50\%$ leaves you at $0.75\times$, not back to even. What compounds is the product of gross returns, so the quantity that matters is the average log return — the geometric growth rate: $$g = \mathbb{E}\!\left[\ln\frac{W_{t+1}}{W_t}\right].$$ Maximizing expected arithmetic wealth tells you to bet everything on the smallest edge (and eventually go broke); maximizing expected log wealth is what maximizes long-run compounded growth. That single choice of objective is the Kelly criterion.

Worked example 1 (why log). Bet your whole bankroll repeatedly on a $+50\%$/$-40\%$ even-chance gamble. The arithmetic mean per round is $+5\%$ — looks great. But the median path multiplies by $\sqrt{1.5\cdot0.6}=0.95$ each round — you lose ~5% of your capital per round and go broke. Log wealth, not arithmetic wealth, tells the truth.

13.3 The Kelly fraction for a simple bet

Bet a fraction $f$ of your bankroll on a wager that wins net odds $b$ with probability $p$ (and loses the stake with probability $q=1-p$). Expected log growth is $$g(f) = p\ln(1+bf) + q\ln(1-f).$$ Set $g'(f)=0$: $$\frac{pb}{1+bf} - \frac{q}{1-f} = 0 \;\Longrightarrow\; \boxed{\ f^\* = \frac{pb-q}{b} = p - \frac{q}{b}\ }$$ which reads beautifully as $f^\*=\dfrac{\text{edge}}{\text{odds}}$. For an even-money bet ($b=1$) it collapses to $f^\*=2p-1$ — just twice your probability edge over a coin.

Worked example 2 (biased coin). A $55\%$ edge at even money: $f^\*=2(0.55)-1=0.10$ — bet 10% of the bankroll. Not 55%, not "all in on a winner" — 10%.

Worked example 3 (better than even money). $p=0.6$, net odds $b=2$: $f^\*=(0.6\cdot2-0.4)/2=0.8/2=0.40$. The bigger payout justifies a bigger fraction, but still far from everything.

13.4 Kelly for a continuous strategy — the μ/σ² rule

A trade isn't a coin flip; it has a mean return $\mu$ and variance $\sigma^2$. Expanding the log-growth to second order (a Taylor step — Chapter 2) gives $$g(f) \approx f\mu - \tfrac12 f^2\sigma^2,$$ a downward parabola in $f$. Maximizing: $$\boxed{\ f^\* = \frac{\mu}{\sigma^2}\ },\qquad g^\* = \frac{\mu^2}{2\sigma^2} = \frac{\text{SR}^2}{2}.$$ Two gifts here. First, optimal size is mean over variance — reward per unit of risk-squared. Second, the best achievable growth rate is half the Sharpe squared — the single cleanest reason the Sharpe ratio is the number: it is your compounding speed.

Worked example 4 (growth = SR²/2). A strategy with per-trade $\mu=0.05$ and $\sigma=0.20$ has $f^\*=0.05/0.04=1.25$ (i.e. 1.25× leverage) and Sharpe $0.25$, so $g^\*=0.25^2/2=0.03125$ log units per trade. Full Kelly here is levered — which is exactly why the next section matters.

13.5 Why full Kelly is too aggressive

Kelly maximizes growth but says nothing kind about the ride. At full Kelly the expected drawdown is savage — a 50% drawdown is roughly a coin-flip event over a long horizon — and the wealth path is wildly volatile. Worse, the formula assumes you know $p$, $\mu$, $\sigma$ exactly. You don't (Chapter 12): you have noisy estimates, and the growth curve is asymmetric — overbetting past $f^\*$ costs you growth faster than the same-size underbet. Because of that asymmetry, when in doubt you bet less.

13.6 Fractional Kelly

The standard fix is to bet a fixed fraction of Kelly — commonly half-Kelly. Plug $f=\tfrac12 f^\*$ into the parabola $g(f)=f\mu-\tfrac12 f^2\sigma^2$ with $f^\*=\mu/\sigma^2$ and $g^\*=\mu^2/(2\sigma^2)$: $$g(\tfrac12 f^\*) = \tfrac12 f^\*\mu - \tfrac12(\tfrac12 f^\*)^2\sigma^2 = \tfrac34\,g^\*,$$ so half-Kelly keeps 75% of the growth for roughly half the wealth-volatility and dramatically smaller drawdowns. That trade is why almost no serious practitioner bets full Kelly.

Interactive (rotate / zoom): the log-growth surface $g(f,p)=p\ln(1+f)+(1-p)\ln(1-f)$ over bet fraction and win probability. The ridge running across it is the optimal $f^\*(p)$; the cliff falling to $-\infty$ on the right is where you bet too much and any loss ruins you.

Worked example 5 (overbetting is ruin). Growth is zero at $f=2f^\*$ and negative beyond. Betting $3\times$ Kelly on a real edge has negative expected log-growth — you go broke while being right on average. There is no reward for courage past $2f^\*$; there is only ruin.

13.7 Estimation error → bet a fraction of a fraction

Tie this back to Chapter 12. Your edge estimate has a confidence interval; the true $f^\*$ could be much smaller than the sample says, and an overfit Sharpe implies a Kelly bet on an edge that isn't there. Since overbetting is punished asymmetrically, uncertainty is a reason to shrink the bet — treat fractional Kelly as insurance against your own estimation error, not just against variance. A confident signal on a wide CI still gets a small size.

13.8 Multiple bets and correlation

Real desks hold several positions at once. Optimal sizing across simultaneous bets uses the inverse covariance matrix of their returns, $f^\* \propto \Sigma^{-1}\mu$ — the direct multivariate generalization of $\mu/\sigma^2$, and a callback to Chapter 11: correlated bets are partly the same bet, so the covariance shrinks their combined size. Two highly correlated longs are close to one double-sized long, and Kelly sizes them accordingly.

13.9 Chapter summary

• Wealth compounds multiplicatively → maximize expected log wealth (geometric growth), not arithmetic.
• Discrete Kelly: $f^\* = (pb-q)/b = \text{edge}/\text{odds}$; even money $\Rightarrow 2p-1$.
• Continuous Kelly: $f^\*=\mu/\sigma^2$, with max growth $g^\*=\text{SR}^2/2$ — the Sharpe is your compounding speed.
• Growth is a parabola in $f$: zero at $2f^\*$, negative beyond — overbetting ruins.
• Full Kelly is too wild; half-Kelly keeps ~75% of the growth at far less risk.
• Estimation error (Ch12) is a further reason to bet a fraction of Kelly.
• Correlated simultaneous bets share risk → size via $\Sigma^{-1}\mu$ (Ch11).

Key formulas to carry forward:

Exercises

13.1 (Even money.) A signal wins $58\%$ of the time at even odds. What's the full-Kelly fraction? What's half-Kelly?

13.2 (General odds.) $p=0.5$, net odds $b=3$ (win 3, lose 1). Compute $f^\*$. Why is it positive even though you win only half the time?

13.3 (Continuous.) A trade has $\mu=0.02$, $\sigma=0.10$. Give $f^\*$ and the optimal growth rate $g^\*$.

13.4 (Overbetting.) Using the parabola $g(f)=f\mu-\tfrac12 f^2\sigma^2$ with the numbers from 13.3, at what fraction does growth hit zero? What happens beyond it?

13.5 (Sharpe link.) A strategy has an annual Sharpe of $1.0$. What is its maximum growth rate under full Kelly, in log units per year?

13.6 (Estimation error.) Your backtested edge has a wide confidence interval. Give two reasons to bet well below the Kelly fraction it implies.

Solutions

Symbols & terms in this chapter

Next: Chapter 14 — Capstone: Building & Validating an Honest Model, where every part of the book comes together.

Chapter 14 — Capstone: Building & Validating an Honest Model

Maps to: the whole terminal — lib/openingRange.ts, lib/openBias.ts, lib/scoreBrief.ts, scripts/backtest-*.mjs — seen now as one pipeline. Lab: build a complete end-to-end model — engineer factors, check redundancy, fit, validate out-of-sample, and size with Kelly — the whole book in forty lines.

Learning objectives

By the end you can: (1) assemble the entire book into one end-to-end workflow; (2) run the honest-model checklist on any strategy; (3) recognize the seven deadly sins that turn a good idea into a losing one; (4) defend every estimator and assumption in a model you ship. This chapter builds nothing new — it connects everything.

14.1 The whole book, in one sentence

Frame a question as a probability or a magnitude, estimate it from data without fooling yourself, size the bet to the confidence you actually have, and keep score honestly. Every chapter is one link in that chain. The Desk Read is that chain running every morning at 9:30.

14.2 The pipeline, stage by stage

1 — Frame the question (Ch. 1). Decide exactly what you're predicting and over what horizon: a probability (will NQ be up 5 minutes from now?) or a magnitude (how wide is today's range?). Everything downstream depends on this being a well-posed random variable with a distribution.

2 — Model the data-generating process (Ch. 1, 5, 7). Returns are roughly Normal, prices Lognormal; dispersion scales like $\sqrt{t}$; the path is a random walk with drift and diffusion. Know the fat tails (Ch. 1) before you trust any Normal-based interval.

3 — Engineer factors (Ch. 2, 8, 9, 11). Turn raw market state into predictors: an overnight volatility shift, rates, the dollar, dealer-gamma structure. Keep them orthogonal — a factor spanned by the others (high VIF) adds collinearity, not signal (Ch. 11). This is where econ-surprise was rejected.

4 — Fit (Ch. 4, 6, 10). Least squares for a magnitude ($\hat\beta=(X^\top X)^{-1}X^\top y$), logistic regression for a probability ($p=\text{sigmoid}(\beta\cdot x)$). Standardize factors so coefficients are comparable.

5 — Diagnose (Ch. 3, 6). Read the residuals, respect the $R^2$-only-rises caveat, watch bias–variance, and regularize if the model is over-flexible. A great in-sample fit is a warning, not a trophy.

6 — Validate honestly (Ch. 3, 12). The gate. Put a standard error on every number, an interval on every edge, and correct for multiple testing — the deflated Sharpe, the expected-max-$t$ bar $\sqrt{2\ln N}$. Estimate performance out-of-sample by construction (walk-forward, purged CV). Nothing passes to money until it clears this stage.

7 — Size the bet (Ch. 13). Convert the validated edge into a position with Kelly, $f^\*=\mu/\sigma^2$, then take a fraction of it to respect estimation error. Confidence, not conviction, sets the size.

8 — Ship and keep score (Ch. 3, 10, 12). Track calibration (does 60% mean 60%?) and accuracy against realized outcomes (scoreBrief.ts), never using information from after the call (no look-ahead). And know when not to refit — six good days are not a signal (Ch. 3, 12).

14.3 A worked pipeline: an openBias-style P(up)

Worked example 1 (frame → fit). Predict $P(\text{NQ up over the next 5 min})$ from two standardized factors: an overnight vol shift $x_1$ and a rates move $x_2$. Fit logistic regression by gradient descent (Ch. 10) on training sessions → $\hat\beta=(\beta_0,\beta_1,\beta_2)$. A session with neutral factors gives $z\approx0$, $p\approx0.5$ — the coin flip, exactly as the terminal reports it.

Worked example 2 (does the factor earn its place?). Before adding a third factor, econ-surprise $x_3$, regress it on $x_1,x_2$: $R^2=0.88\Rightarrow\text{VIF}=1/(1-0.88)\approx8.3$. High. And out-of-sample it lifts accuracy by $0.1\%$ — inside the noise. Verdict: reject (Ch. 11) — the real openBias call.

Worked example 3 (validate → size). The kept two-factor model shows an out-of-sample edge equivalent to a daily Sharpe of $0.12$ over $n=250$ sessions: $t\approx0.12\sqrt{250}\approx1.9$ — barely significant, wide CI. Full Kelly from the point estimate would be reckless; the honest size is a small fraction of Kelly, scaled to that thin $t$ (Ch. 12–13). A real-but-uncertain edge gets a real-but-small bet.

14.4 The honest-model checklist

Run this before trusting any model — it is the book, condensed:

14.5 The seven deadly sins

Each is a chapter's warning, restated as a way to lose money:

1. Overfitting — memorizing noise; dazzling in-sample, dead out-of-sample (Ch. 6).
2. Look-ahead bias — using data you wouldn't have had in real time (Ch. 12).
3. Multiple testing — searching until noise looks significant (Ch. 12).
4. Multicollinearity — piling on correlated factors and trusting the coefficients (Ch. 11).
5. Ignoring costs — a paper edge that lives entirely inside the spread (Ch. 12).
6. Overbetting — full-or-more Kelly, ruin despite a real edge (Ch. 13).
7. Refitting on noise — "fixing" the model after a lucky or unlucky streak (Ch. 3).

Interactive (rotate / zoom): the capstone trade-off — expected out-of-sample growth over model complexity (Ch. 6) and bet fraction (Ch. 13). There is a single ridge: enough complexity to capture the signal, enough size to compound it — and two cliffs, overfitting on one axis and overbetting on the other. Honest modeling lives on the ridge.

Worked example 4 (the joint optimum). Add complexity and in-sample fit keeps rising, but out-of-sample edge peaks then falls (bias–variance). Independently, growth peaks at Kelly then falls (overbetting). The best shipped model maximizes the product — moderate complexity sized at fractional Kelly — not either extreme. Both cliffs are failures of the same discipline: respect what you don't know.

14.6 Where this goes next

You can now read every estimator in the Desk Read and say why it's there, what it assumes, and when it lies. That was the goal. From here: the appendices (matrix calculus, the Normal integral, numerical methods, a notation table) fill in machinery; and the real practice is doing this loop on your own NQ data, one honest model at a time.

14.7 Chapter summary — and the book's

• Modeling is a pipeline: frame → model the process → engineer orthogonal factors → fit → diagnose → validate → size → ship & track.
• The validation gate (SEs, CIs, multiple-testing correction, out-of-sample) is what separates an edge from a story.
• Size with fractional Kelly — to your confidence, never your conviction.
• Avoid the seven deadly sins; run the ten-point checklist.
• Every technique in the book is one link: probability and calculus and linear algebra become volatility and options and gamma, become a probability model, become a validated, sized, tracked decision. That chain is the mathematics of the Desk Read.

The one formula to carry out of the book:

Exercises

14.1 (Pipeline order.) A colleague sizes a position with Kelly before running any out-of-sample test. Which stage did they skip, and why is the order fatal?

14.2 (Checklist.) A model shows in-sample $R^2=0.4$ across 30 candidate factor sets, best $t=2.3$, no hold-out. Which checklist items fail?

14.3 (Factor decision.) A new factor has VIF $=9$ and adds $0.05\%$ out-of-sample accuracy. Keep or reject? Which two chapters decide it?

14.4 (Edge to size.) An out-of-sample edge is a Sharpe of $0.10$ with a 95% CI of $[-0.05, 0.25]$. Argue for a small fraction of Kelly in one sentence.

14.5 (Name the sin.) (a) "I retrained after three losing days." (b) "The backtest used the day's high to time the entry." (c) "I kept the best of 40 parameter sets." Identify each.

14.6 (Defense.) Pick any estimator in the Desk Read (range coefficients, $P(\text{up})$, GEX walls) and, in a sentence, state its assumption and the chapter that justifies it.

Solutions

Symbols & terms in this chapter

This completes The Mathematics of the Desk Read. Every morning's brief is now a chain of derivations you can read end to end — and defend. The appendices follow; the rest is practice.

Appendix A — Matrix Calculus & Vector Derivatives

Why this appendix: Chapters 4, 6, 10, 11 and 13 all lean on differentiating scalar functions of vectors — the normal equations, the logistic gradient, ridge, Kelly across correlated bets. Here is the small toolkit that makes those derivations mechanical instead of mysterious.

A.1 The gradient — a derivative that points

For a scalar function $f(\mathbf{x})$ of a vector $\mathbf{x}=(x_1,\dots,x_n)^\top$, the gradient is the column vector of partial derivatives: $$\nabla f = \frac{\partial f}{\partial\mathbf{x}} = \Big(\frac{\partial f}{\partial x_1},\dots,\frac{\partial f}{\partial x_n}\Big)^\top.$$ It points in the direction of steepest increase, and it is zero at a minimum or maximum — the fact every fitting routine in the book exploits. Gradient descent (Ch. 10, 13) simply steps against it.

A.2 The four identities you actually need

Let $\mathbf{a}$ be a constant vector, $A$ a constant matrix, $\mathbf{x}$ the variable. These cover almost every derivation in the book:

They are just the scalar rules ($\frac{d}{dx}(ax)=a$, $\frac{d}{dx}(ax^2)=2ax$) wearing vector clothes. The only trap is the transpose in the general quadratic; when $A$ is symmetric — as covariance matrices and $X^\top X$ always are — it collapses to the clean $2A\mathbf{x}$.

A.3 Deriving the normal equations (Chapter 4, in full)

Least squares minimizes the squared residual $S(\boldsymbol\beta)=\lVert\mathbf{y}-X\boldsymbol\beta\rVert^2$. Expand: $$S = (\mathbf{y}-X\boldsymbol\beta)^\top(\mathbf{y}-X\boldsymbol\beta) = \mathbf{y}^\top\mathbf{y} - 2\boldsymbol\beta^\top X^\top\mathbf{y} + \boldsymbol\beta^\top X^\top X\boldsymbol\beta.$$ Differentiate term by term with A.2 (note $X^\top X$ is symmetric): $$\frac{\partial S}{\partial\boldsymbol\beta} = -2X^\top\mathbf{y} + 2X^\top X\boldsymbol\beta.$$ Set to zero: $$\boxed{\ X^\top X\,\hat{\boldsymbol\beta} = X^\top\mathbf{y}\ \Longrightarrow\ \hat{\boldsymbol\beta}=(X^\top X)^{-1}X^\top\mathbf{y}.\ }$$ That is the entire OLS derivation — three identities and one "set the gradient to zero." Ridge regression (Ch. 6) adds $\lambda\lVert\boldsymbol\beta\rVert^2$, whose derivative $2\lambda\boldsymbol\beta$ gives $\hat{\boldsymbol\beta}=(X^\top X+\lambda I)^{-1}X^\top\mathbf{y}$ — the same machinery, one extra term.

Worked example A.1 (the logistic gradient). The log-loss is $\mathcal L=-\sum_i[y_i\ln p_i+(1-y_i)\ln(1-p_i)]$ with $p_i=\text{sigmoid}(\mathbf{x}_i^\top\boldsymbol\beta)$. Using the tidy fact $\frac{d}{dz}\text{sigmoid}(z) > = p(1-p)$, the chain rule collapses the whole thing to $\nabla\mathcal L=\sum_i(p_i-y_i)\mathbf{x}_i = X^\top(\mathbf{p}-\mathbf{y})$ — residual times feature, exactly the Chapter 10 gradient. The messy $p(1-p)$ terms cancel; that cancellation is why logistic regression is fit with such a clean update.

A.4 The Hessian — curvature in many dimensions

The second derivative of $f(\mathbf{x})$ is the Hessian matrix $H$ with $H_{ij}=\partial^2 f/\partial x_i\partial x_j$. It is the multivariate version of Chapter 2's gamma. Its role here: a critical point is a minimum when $H$ is positive-definite (curves up in every direction). For least squares, $H=2X^\top X$ — positive-definite exactly when $X$ has full column rank, i.e. no perfect collinearity (Chapter 11). When columns are nearly collinear, $H$ is nearly singular, the bottom of the bowl flattens into a ravine, and $\hat{\boldsymbol\beta}$ becomes unstable — the same story told three ways across the book.

A.5 The chain rule, vector form

For a composition, gradients multiply by Jacobians (the matrix of all first partials). In the scalar-output case this reads $\nabla_{\mathbf{x}} f = J^\top\,\nabla_{\mathbf{u}} f$, where $\mathbf{u}=g(\mathbf{x})$ and $J$ is the Jacobian of $g$. Every backpropagation and every "differentiate the loss w.r.t. the parameters" is this rule applied repeatedly — including the logistic gradient of A.3.

A.6 Quick reference

Continue to Appendix B — The Gaussian Integral & Normal Facts.

Appendix B — The Gaussian Integral & Normal Facts

Why this appendix: the Normal distribution is the spine of the book — Chapters 1, 3, 5, 7 and 8 all rest on it. Two questions get asked over and over: why does the bell curve integrate to one, and what are the numbers I keep reusing. Here are both, with the one integral trick worth seeing once in your life.

B.1 The Gaussian integral

Everything Normal depends on this single result: $$\boxed{\ \int_{-\infty}^{\infty} e^{-x^2}\,dx = \sqrt{\pi}.\ }$$ You cannot get it by finding an antiderivative — $e^{-x^2}$ has none in elementary terms. The trick is to compute the square of the integral in two dimensions and switch to polar coordinates. Let $I=\int_{-\infty}^\infty e^{-x^2}dx$. Then $$I^2 = \int_{-\infty}^\infty e^{-x^2}dx\int_{-\infty}^\infty e^{-y^2}dy = \iint_{\mathbb R^2} e^{-(x^2+y^2)}\,dx\,dy.$$ Switch to polar ($x^2+y^2=r^2$, $dx\,dy=r\,dr\,d\theta$) — and now there is an antiderivative, because the stray $r$ is exactly what $\frac{d}{dr}e^{-r^2}$ needs: $$I^2 = \int_0^{2\pi}\!\!\int_0^\infty e^{-r^2}\,r\,dr\,d\theta = 2\pi\cdot\Big[-\tfrac12 e^{-r^2}\Big]_0^\infty = 2\pi\cdot\tfrac12 = \pi.$$ So $I=\sqrt\pi$. Squaring turned an impossible 1-D integral into an easy 2-D one — a trick worth remembering far beyond this book.

B.2 Why the Normal density integrates to one

Rescale. The standard Normal density is $\varphi(z)=\frac{1}{\sqrt{2\pi}}e^{-z^2/2}$. Substitute $x=z/\sqrt2$ (so $z^2/2=x^2$, $dz=\sqrt2\,dx$) into B.1: $$\int_{-\infty}^\infty e^{-z^2/2}\,dz = \sqrt2\int_{-\infty}^\infty e^{-x^2}\,dx = \sqrt2\cdot\sqrt\pi = \sqrt{2\pi}.$$ Divide by $\sqrt{2\pi}$ and the total probability is exactly $1$. The mysterious $\sqrt{2\pi}$ in front of every Normal density is there for exactly one reason: to cancel this integral and make the area one. The general $\mathcal N(\mu,\sigma^2)$ follows by shifting ($z=(x-\mu)/\sigma$) and scaling, which contributes the $1/\sigma$.

B.3 Moments, and why odd ones vanish

The mean and variance come from the same Gaussian machinery: $$\mathbb E[Z]=\int z\,\varphi(z)\,dz = 0,\qquad \text{Var}(Z)=\mathbb E[Z^2]=\int z^2\varphi(z)\,dz = 1.$$ The mean is zero because $z\,\varphi(z)$ is an odd function integrated over a symmetric range — every positive $z$ cancels its negative twin. That symmetry is why a driftless random walk (Ch. 7) has expected displacement zero while its variance still grows. The $\mathbb E[Z^2]=1$ integral is B.1 with an extra $z^2$, evaluated by parts.

B.4 The numbers you keep reusing

B.5 The error function — how the CDF is actually computed

There is no closed form for $\Phi(z)=\int_{-\infty}^z\varphi$, so software uses the error function $\text{erf}$, a rescaled Gaussian integral, via the identity in the table. This is exactly the trick the book's labs use — since Pyodide ships no SciPy, the Normal CDF is built as 0.5*(1+math.erf(z/sqrt(2))). Black–Scholes' $N(d_1)$, a confidence interval's tail, a p-value — all are one erf call underneath.

Worked example B.1 (a band probability from erf). $P(-1<Z<1)=\Phi(1)-\Phi(-1)=[\tfrac12(1+\text{erf}(1/\sqrt2))]-[\tfrac12(1-\text{erf}(1/\sqrt2))] > =\text{erf}(1/\sqrt2)=\text{erf}(0.707)\approx0.683$ — the 68% in 68–95–99.7, straight from the error function.

B.6 Quick reference

• The trick: square the 1-D integral, go polar, the Jacobian $r$ makes it integrable → $\int e^{-x^2}=\sqrt\pi$.
• The constant: $\sqrt{2\pi}$ exists solely to normalize the Normal to area one.
• Symmetry: odd moments vanish → zero mean, $\Phi(-z)=1-\Phi(z)$.
• In code: $\Phi(z)=\tfrac12(1+\text{erf}(z/\sqrt2))$ — the whole Normal CDF in one line.

Continue to Appendix C — Numerical Methods.

Appendix C — Numerical Methods

Why this appendix: several quantities in the book have no closed form — implied volatility, a fitted logistic, an option price under a general model. This appendix collects the four numerical workhorses the labs use, with the one thing you must know about each: how fast it converges and how it can fail.

C.1 Finite differences — derivatives without calculus

To estimate a derivative you don't have in closed form, perturb the input and measure the change. The central difference is the one to use: $$f'(x)\approx\frac{f(x+h)-f(x-h)}{2h},\qquad f''(x)\approx\frac{f(x+h)-2f(x)+f(x-h)}{h^2}.$$ The central form has error $O(h^2)$ — halving $h$ cuts the error fourfold — versus $O(h)$ for the naive one-sided $\frac{f(x+h)-f(x)}{h}$. This is exactly how the Chapter 2 lab computes delta and gamma and checks them against the analytic Greeks.

The catch is a tension: too large an $h$ and the $O(h^2)$ truncation error dominates; too small and floating-point cancellation (subtracting two nearly equal numbers) wrecks precision. A sweet spot near $h\approx\sqrt[3]{\varepsilon}\,x$ (with $\varepsilon\approx10^{-16}$ machine precision) balances the two — in practice $h\sim10^{-5}$ for a typical scale.

Worked example C.1 (why central beats forward). For $f(x)=e^x$ at $x=0$ with $h=0.01$: forward difference gives $\frac{e^{0.01}-1}{0.01}\approx1.00501$ (error $5\times10^{-3}$); central gives $\frac{e^{0.01}-e^{-0.01}}{0.02}\approx1.0000167$ (error $1.7\times10^{-5}$). Same $h$, ~300× more accurate — for the cost of one extra evaluation.

C.2 Root finding — solving $g(x)=0$

Implied volatility, break-evens, the gamma-flip level — all are roots of a function with no algebraic inverse.

• Bisection. Bracket a sign change $[a,b]$ and halve it repeatedly. Dead reliable, but slow — linear convergence, one bit per step (~4 steps per decimal digit).
• Newton's method. Step $x_{n+1}=x_n-\dfrac{g(x_n)}{g'(x_n)}$. Quadratic convergence — the number of correct digits roughly doubles each step — but it needs the derivative and can diverge from a bad start.
• Practice. Robust solvers (Brent's method) bracket like bisection but accelerate like Newton — reliability and speed. Implied vol is typically solved this way: find the $\sigma$ making Black–Scholes match the market price.

C.3 Monte Carlo — integration by sampling

When a probability or expectation is a high-dimensional integral, sample it: draw $N$ random scenarios, average the payoff. The error shrinks like $$\text{error}\sim\frac{\sigma}{\sqrt N},$$ the $1/\sqrt N$ rule of Chapter 7 — the same $\sqrt N$ from the standard error of a mean (Ch. 3). Its blessing and curse: the rate is independent of dimension (great for many-asset problems) but slow — 100× more samples for one more digit of accuracy. Variance-reduction tricks (antithetic variates, control variates) buy accuracy without more samples by shrinking the $\sigma$ in the numerator. The Chapter 7 lab prices a call this way and watches the estimate converge.

Worked example C.2 (how many paths?). To halve a Monte Carlo error you need $4\times$ the paths; for one more decimal digit ($10\times$ tighter), $100\times$. A price estimated to $\pm0.5$ with 10,000 paths needs ~1,000,000 for $\pm0.05$. This is why variance reduction matters.

C.4 Gradient descent — optimization by stepping downhill

For a loss with no closed-form minimum — logistic regression (Ch. 10), Kelly-style objectives (Ch. 13) — step against the gradient (Appendix A): $$\boldsymbol\beta \leftarrow \boldsymbol\beta - \eta\,\nabla\mathcal L(\boldsymbol\beta).$$ The learning rate $\eta$ is everything: too small and it crawls; too large and it oscillates or diverges. Two practical notes tie back to the book:

• Standardize the features (Ch. 10). If factors are on wildly different scales the loss surface is a stretched ravine and descent zig-zags — the same ill-conditioning as collinearity (Ch. 11), now slowing the optimizer.
• Convergence. On a convex loss (logistic log-loss is convex) a reasonable $\eta$ is guaranteed to reach the global minimum; the condition number of the Hessian $X^\top X$ sets how fast.

C.5 A note on floating point

Computers store ~16 significant decimal digits. Two hazards recur: cancellation (subtracting near-equal numbers, as in C.1) and overflow in $e^z$ for large $z$ — which is why a numerically careful sigmoid computes $1/(1+e^{-z})$ for $z\ge0$ and $e^z/(1+e^z)$ for $z<0$, never exponentiating a large positive number. Small defensive rewrites like this keep the labs stable.

C.6 Quick reference

Continue to Appendix D — Notation & Symbols.

Appendix D — Notation & Symbols

Why this appendix: every chapter closes with its own symbol table; this is the master index — one place to look up any symbol in the book, plus the conventions that hold throughout. When a formula in Chapter 8 reuses a letter from Chapter 3, this table settles what it means.

D.1 Conventions used throughout

• Vectors are lower-case bold ($\mathbf{x}$), matrices upper-case ($X$), scalars italic ($x$).
• A hat means "estimated from data" ($\hat\beta$, $\hat\sigma$) versus the true value ($\beta$, $\sigma$).
• A bar means a sample average ($\bar r$); $\mathbb E[\cdot]$ is the true (population) expectation.
• Greek letters are reserved for parameters and population quantities; Latin for data and estimates.
• $\ln$ is natural log (base $e$) everywhere — never base 10.
• Time is in the natural unit of the context (minutes for the 5-min move, years for options); $\sqrt{t}$ scaling (Ch. 5) converts between them.

D.2 Master symbol table

D.3 The recurring themes, in symbols

A handful of ideas wear different notation in different chapters but are the same idea:

Spotting these equivalences is the mathematical maturity the book was building toward — the same structure, seen from wherever the Desk Read happens to need it.

This completes the appendices, and The Mathematics of the Desk Read. Every symbol in the morning brief now has a home, a meaning, and a derivation you can follow end to end.