What's the one-line summary?

QDA on the same UK Core CPI × GBPCHF 6-window panel Stats #34 (MANOVA), #35 (canonical direction), #36 (LDA), and #37 (Mahalanobis) used. Same n=196, K=5 buckets. QDA gives each class its OWN 6×6 covariance matrix Σ_k instead of LDA's pooled Σ. Full-sample: QDA 57.14% (LDA 49.49%, +7.65pp better in-sample). LOO: QDA 40.31% (LDA 42.86%, -2.55pp WORSE out-of-sample). QDA's train→test gap is 16.83pp vs LDA's 6.63pp. Classic bias-variance overfitting: QDA has 2.65× LDA's parameters (135 vs 51), and the tail-bucket sample sizes (n=13 big_miss, n=15 big_beat) are too small for reliable per-class covariance estimation (13/21 = 0.62 obs/param, well below the 10:1 rule of thumb). Only bucket QDA improves: big_beat (+13.34pp recall) at cost of small_beat (-11.32pp). Regularized QDA (reg_param=0.20) recovers to 42.35% — essentially matches LDA — telling us the pooled Σ already has all the discriminant information.

How is QDA different from LDA (Stats #36)?

The difference is a single modeling choice: whether the class-conditional densities share a covariance matrix. LDA assumes all K classes have the SAME covariance Σ, estimates one pooled Σ from the whole dataset (weighted by class size), and produces LINEAR decision boundaries between classes. QDA relaxes that assumption — each class k gets its OWN covariance Σ_k, producing QUADRATIC decision boundaries. Mathematically, the discriminant function becomes δ_k(x) = -½ (x-μ_k)ᵀ Σ_k⁻¹ (x-μ_k) - ½ log|Σ_k| + log π_k (with the log|Σ_k| term specific to QDA). In theory, QDA is more flexible and should do at least as well as LDA when class covariances differ. In practice — as this dataset shows — the extra flexibility costs more variance than it saves in bias, and out-of-sample accuracy drops.

Why does QDA overfit here even though it 'should' be better?

Parameter count. QDA has 2.65× the free parameters of LDA. LDA: 5 means × 6 dims + 1 pooled covariance (6×7/2 = 21 free) = 51 total parameters. QDA: 5 means × 6 dims + 5 covariances × 21 = 135 total parameters. The ratio of observations to parameters is 196/51 = 3.8 for LDA and 196/135 = 1.5 for QDA — both below the 10:1 rule of thumb, but QDA's ratio is 2.5× worse. Worse, for the small tail buckets, QDA has to estimate a 21-parameter covariance from n=13 or n=15 observations, giving 0.62 and 0.71 obs/param respectively — an order of magnitude below the classical threshold. Per-class covariance condition numbers confirm the diagnosis: big_miss cov cond# = 462, big_beat = 525 (both ~3× worse than mid-bucket cov cond# ≈ 155). Ill-conditioned covariance matrices inflate Mahalanobis distances in eigen-directions where the tail-bucket sample happens to have low variance, causing spurious classifications AWAY from those buckets. Net effect: QDA loses out-of-sample accuracy despite gaining in-sample.

Does QDA at least improve tail-bucket recall (the LDA weakness Stats #36 flagged)?

Partially — for big_beat only. QDA per-class LOO recall vs LDA per-class LOO recall: big_miss 0/13 (both zero — log-prior handicap dominates), small_miss 24/53 = 45.28% vs LDA 49.06% (-3.78pp), in_line 31/62 = 50.00% vs 48.39% (+1.61pp), small_beat 20/53 = 37.74% vs 49.06% (-11.32pp), big_beat 4/15 = 26.67% vs 13.33% (+13.34pp). Only big_beat sees a QDA improvement (+13.34pp). QDA gives big_beat its own covariance ellipsoid that captures the fatter tails than pooled Σ would recognize. But this comes at the cost of small_beat losing 11.32pp (49.06 → 37.74) — the small_beat bucket's mid-region gets partly reclassified as big_beat because QDA's expanded big_beat likelihood surface bleeds into small_beat territory. Net LOO across all buckets: QDA -2.55pp vs LDA. And big_miss stays at 0/13 because the empirical prior log(13/196) = -2.71 handicap doesn't go away just because the class density model changed.

What's the practical lesson?

Relaxing model assumptions is not automatically an improvement. QDA is more flexible than LDA in theory but on realistic small-sample bucket-classification problems it OVERFITS. The correct next step from LDA when you want more flexibility is not 'more parameters per class' — it's SHRINKAGE (regularization) toward the pooled model, or a SMALLER feature set (e.g., drop from d=6 windows to just 15m/1h — halves the number of covariance parameters). Complexity for its own sake makes out-of-sample worse. For traders building bucket-classification models: default to LDA over QDA when your tail-bucket sample sizes are below ~10× the number of features. If you already have enough data for QDA to work (rule of thumb: n_class ≥ 10 × d(d+1)/2 per class = 10 × 21 = 210 observations per class for d=6), then QDA can beat LDA. But most macro-news bucket classifications, like this one, don't have that luxury.

← All insights

Statistics7 September 2026· 6 min read

Statistics for Traders #38: QDA on the same UK Core CPI × GBPCHF 6-window response panel Stats #34/#35/#36/#37 used. QDA UNDERPERFORMS LDA on LOO: 79/196 = 40.31% vs LDA's 84/196 = 42.86% (-2.55pp), despite QDA fitting 57.14% in-sample vs LDA's 49.49%. Bias-variance overfitting — QDA's 2.65× extra parameters (135 vs 51) can't be reliably estimated from small tail-bucket samples (n=13 / n=15). Regularized QDA (reg_param=0.20) recovers to LDA-parity 42.35% — confirming the pooled covariance already has essentially all the discriminant information.

Q: Regularized QDA — is that the fix?

Partially. sklearn QDA(reg_param=λ) regularizes each per-class covariance toward the identity: Σ_k → (1-λ)·Σ_k + λ·I. Sweep across λ: reg=0.00 → 40.31%, reg=0.05/0.10/0.15 → 41.84%, reg=0.20 → 42.35% (best), reg=0.30 → 40.82%, reg=0.50 → 40.82%, reg=0.70 → 40.31%, reg=1.00 → 32.65% (each Σ_k replaced by identity → near-random). The best regularization (reg_param=0.20) gets QDA to 42.35% — nearly matches LDA's 42.86% but doesn't beat it. This tells us: on this dataset, the pooled Σ has essentially all the discriminant information; per-class Σ_k structure adds nothing meaningful once its noise is controlled. If the true class covariances DID differ substantially, regularized QDA would beat LDA. The natural next step from here is Friedman's (1989) Regularized Discriminant Analysis (RDA), which regularizes each Σ_k toward the pooled Σ AND toward the identity, with cross-validation to pick the right shrinkage — that's a good follow-up post.

Statistics for Traders #38. QDA (Quadratic Discriminant Analysis) on the SAME UK Core CPI × GBPCHF 6-window response panel that Stats #34 (MANOVA), Stats #35 (canonical direction), Stats #36 (LDA), and Stats #37 (Mahalanobis) used. n=196, K=5, d=6. QDA gives each class its OWN 6×6 covariance matrix Σ_k instead of LDA’s pooled Σ.

Headline result: QDA underperforms LDA on out-of-sample LOO despite fitting 7.65pp better in-sample. Full-sample: QDA 112/196 = 57.14% vs LDA 97/196 = 49.49%. LOO: QDA 79/196 = 40.31% vs LDA 84/196 = 42.86%. QDA’s train→test generalization gap is 16.83pp — 2.54× the LDA gap of 6.63pp. Classic bias-variance overfitting story.

Three-panel chart. Top panel: bar chart comparing full-sample and LOO accuracy for QDA (57.14% full, 40.31% LOO) vs LDA (49.49% full, 42.86% LOO) vs majority baseline (31.63%) vs QDA regularized (42.35%) vs QDA uniform priors (35.20%). QDA train→test gap is 16.8pp vs LDA's 6.6pp. Middle panel: parameter-count table showing LDA has 51 total parameters (30 means + 21 pooled cov), QDA has 135 (30 means + 5 covariances × 21 each), a 2.65× ratio; and highlighting that fitting the big_miss covariance alone means 21 parameters from 13 observations, or 0.62 obs/param. Bottom panel: per-class LOO recall bars for QDA (coral) vs LDA (green) across the 5 buckets — big_miss 0/13 for both, small_miss 45.28 vs 49.06%, in_line 50.00 vs 48.39%, small_beat 37.74 vs 49.06% (QDA loses 11pp), big_beat 26.67 vs 13.33% (QDA gains 13pp). — QDA vs LDA on the UK Core CPI × GBPCHF panel. QDA fits better in-sample (top panel) but generalizes worse in LOO. The parameter-count table (middle) explains why — QDA has 2.65× the free parameters of LDA, and per-class Σ_k needs more observations to estimate than the tail buckets provide. Per-class recall (bottom) shows QDA gains only on big_beat (+13pp) while losing on small_beat (-11pp); net LOO -2.55pp.

The single modeling choice — pooled Σ vs per-class Σ_k

LDA and QDA differ in exactly one assumption: whether the K classes share a covariance matrix. LDA assumes all classes have the SAME Σ, estimates one pooled Σ, and produces LINEAR decision boundaries. QDA relaxes that assumption — each class gets its OWN Σ_k, producing QUADRATIC decision boundaries. The QDA discriminant function is

δ_k(x) = −½ (x−μ_k)ᵀ Σ_k⁻¹ (x−μ_k) − ½ log|Σ_k| + log π_k

with the −½ log|Σ_k| term specific to QDA (LDA has a common −½ log|Σ| that cancels across classes). In theory QDA is strictly more flexible and should do at least as well as LDA when class covariances differ. In practice, on this dataset, the extra flexibility costs more variance than it saves in bias.

Why QDA overfits — parameter counts and observations-per-parameter

Model	Means (K×d)	Covariance	Total params	n / params	Note
LDA	30	21 (pooled)	51	3.8	already below the 10:1 rule of thumb
QDA	30	5 × 21 = 105	135	1.5	2.65× LDA parameters
QDA (big_miss cov alone)	—	21	—	13/21 = 0.62	far below 10:1 — pure noise
QDA (big_beat cov alone)	—	21	—	15/21 = 0.71	far below 10:1 — pure noise

Per-class covariance conditioning confirms the diagnosis: the big_miss covariance matrix has condition number 462 and big_beat has 525 — both roughly 3× the mid-bucket conditioning numbers (~155). Ill-conditioned Σ_k inflates Mahalanobis distances in eigen-directions where the tail-bucket sample happens to have low variance, causing spurious classifications AWAY from those buckets.

Per-class LOO recall — QDA gains big_beat, loses small_beat

Bucket	n	LDA (Stats #36)	QDA (today)	Δ	Reading
big_miss	13	0/13 = 0.00%	0/13 = 0.00%	0.00pp	log-prior handicap dominates in both
small_miss	53	26/53 = 49.06%	24/53 = 45.28%	-3.78pp	slight loss
in_line	62	30/62 = 48.39%	31/62 = 50.00%	+1.61pp	slight gain
small_beat	53	26/53 = 49.06%	20/53 = 37.74%	-11.32pp	biggest loss — QDA's expanded big_beat bleeds into small_beat
big_beat	15	2/15 = 13.33%	4/15 = 26.67%	+13.34pp	biggest gain — QDA captures fat tails pooled Σ misses

Only big_beat sees a QDA improvement. QDA gives big_beat its own covariance ellipsoid that captures the fatter tails than pooled Σ recognizes — the +13.34pp recall gain. But this comes at the cost of small_beat losing -11.32pp: the mid-region of small_beat gets partly reclassified as big_beat because QDA’s expanded big_beat likelihood surface bleeds into small_beat territory. Net LOO across all 5 buckets: -2.55pp.

Regularization rescues QDA — up to LDA-parity, not past it

reg_param λ	LOO acc	Note
0.00	40.31%	unregularized QDA
0.05	41.84%
0.10	41.84%
0.15	41.84%
0.20	42.35%	sweet spot — nearly matches LDA
0.30	40.82%
0.50	40.82%
0.70	40.31%
1.00	32.65%	each Σ_k replaced by identity — near-random

sklearn regularization Σ_k → (1-λ)·Σ_k + λ·I. Best λ is 0.20 and gets QDA to 42.35% — nearly matches LDA’s 42.86%but doesn’t beat it. This tells us: on this dataset, the pooled Σ has essentially all the discriminant information; per-class Σ_k structure adds nothing meaningful once its noise is controlled.

Cross-verification

All numbers cross-verified against sklearn 1.9.0 QuadraticDiscriminantAnalysis(reg_param=0.0) with 100% agreement on both full-sample and LOO predictions. LDA baseline reproduces the Stats #36 result exactly (84/196 = 42.86%). Panel construction (n=196 releases, intersection of all 6 windows) matches Stats #34/#35/#36/#37 exactly — bucket counts big_miss=13, small_miss=53, in_line=62, small_beat=53, big_beat=15 identical.

Practical lesson

Relaxing model assumptions is not automatically an improvement. QDA is more flexible than LDA in theory but on realistic small- sample bucket-classification problems it OVERFITS. The correct next step from LDA when you want more flexibility is not “more parameters per class” but SHRINKAGE toward the pooled model (Friedman’s 1989 Regularized Discriminant Analysis, RDA — a natural follow-up), or a SMALLER feature set (drop from d=6 windows to just 15m/1h — halves the number of covariance parameters). For traders building bucket-classification models: default to LDA over QDA when your tail-bucket sample sizes are below ~10× the number of features. Most macro-news bucket classifications, like this one, don’t have that luxury.

Keep reading