Statistics for Traders #3: why the big_miss bucket's +12 pip median could actually be +105 or −24
A bucket median is an estimate of an unknown true median. How precise the estimate is depends almost entirely on n. On NFP on EURUSD at 15 minutes, the in_line bucket has n=87 and a 95% confidence interval on the median of 16 pips. Same event, same instrument, same window — big_miss has n=10 and a confidence interval of 130 pips.
This is the third post in the Statistics for Traders series. Stats #1 introduced the z-score for sizing the surprise itself. Stats #2 introduced the standard deviation of the move distribution. This one covers n — the reason the tool faints out bars belown = 30 and why that matters.
The demonstration
Every NFP release since 2010 on EURUSD, split into the five surprise buckets, measured at 15 minutes after release. All five buckets are pulled from the same /api/v1/news-impact/releasesendpoint the tool uses. Here’s the median for each bucket, with a 95%+ distribution-free confidence interval attached:
![Horizontal error-bar chart. Five rows, one per NFP surprise bucket on EURUSD 15m. Each row shows the bucket's median move as a coral dot with the 95%+ distribution-free CI shown as a horizontal muted bar behind it, spanning the CI's low and high bounds. Rows from top to bottom: big_miss n=10 median +11.65 CI [-24.3, +105.5] width 130p; small_miss n=37 median +12.10 CI [+4.9, +35.8] width 31p; in_line n=87 median -2.70 CI [-9.7, +6.2] width 16p; small_beat n=47 median -28.50 CI [-35.0, -10.6] width 24p; big_beat n=14 median -34.55 CI [-63.8, -13.6] width 50p. big_miss and big_beat CIs are visibly wider than the middle three buckets. big_miss CI comfortably crosses zero. small_miss CI is entirely on the positive side. in_line and small_beat CIs are narrow bands not far from their medians.](/insights/stats-for-traders-sample-size/median-cis.png)
| Bucket | n | Median | CI lo | CI hi | Width |
|---|---|---|---|---|---|
| big_miss | 10 | +11.65 | −24.30 | +105.50 | 130p |
| small_miss | 37 | +12.10 | +4.90 | +35.80 | 31p |
| in_line | 87 | −2.70 | −9.70 | +6.20 | 16p |
| small_beat | 47 | −28.50 | −35.00 | −10.60 | 24p |
| big_beat | 14 | −34.55 | −63.80 | −13.60 | 50p |
Look at the two miss-side buckets in the table. Their point estimates are almost identical: +11.65 and +12.10. But the confidence intervals are wildly different: 130 pips wide for big_miss and 31 pips wide for small_miss. If you saw both bars on a tool with no nnext to them, you’d read them as equally trustworthy. They’re not.
How the CI is built (no assumptions)
The confidence interval used above is distribution-free — it doesn’t assume the underlying distribution is normal, or symmetric, or anything else. It works directly from the sorted sample, which is a good property because FX move distributions have fat tails and can be skewed, especially in tail buckets.
The recipe. Given a sorted sample of size n, find the largest k such that:
1 − 2 × P(X ≤ k − 1) ≥ 0.95, where X ~ Binomial(n, 0.5)
Then the CI is [x(k), x(n-k+1)] — the k-th smallest and k-th largest values in the sample. For n=10 the largest such k is 2: the CI is the 2nd-smallest and 9th-smallest values. For n=87it’s 34: the CI is the 34th-smallest and 54th-smallest — much closer to the median.
The intuition: with only 10 observations you have to include almost the entire sample to be 95% sure you’ve captured the true median, so the CI stretches from near-min to near-max. With 87 you can trim off 33 values from each side and still be confident.
How wide is my CI going to be? (rule of thumb)
The CI width shrinks roughly as 1 / √n. So:
- doubling n shrinks the CI by about 30% (1/√2 ≈ 0.71);
- quadrupling n halves it;
- to halve the current 130-pip big_miss CI, you’d need roughly 30 more big-miss NFP prints — call it 30 more years of NFP releases.
In other words, for a tail bucket, the estimate you have now is close to the best estimate you’re ever going to have. The way to fix small-n uncertainty is almost never“wait for more data” — it’s “design a bucketing that pulls in more observations,” e.g. by relaxing the surprise-z cutoff, combining adjacent windows, or accepting that the tail bucket signal is inherently loose.
Practical reading rule
A useful rule of thumb when you look at a bucket median: check whether the CI comfortably excludes zero.
On small_miss in this table the CI is [+4.90, +35.80]— entirely positive, so “the true median move is positive” is a claim the sample supports at 95%. On big_miss the CI is [−24.30, +105.50] — crosses zero — so even the direction of the effect is not something the sample supports at 95%. The +11.65point estimate is still your best guess, but you shouldn’t claim you know the sign.
The same rule applies to comparing two buckets. small_beat (CI [−35, −11]) and big_beat (CI [−64, −14]) both sit entirely below zero, so “a beat moves EURUSD down” is supported for both. But the two CIs overlap heavily, so “big_beat moves it more than small_beat” is notsupported at 95% by this sample. That’s the sort of read you can only make once you know how wide the intervals are.
Why the tool uses opacity for this
News Impact Explorer draws bars for small-n buckets at reduced opacity (below n=30). That’s the visual encoding of exactly this uncertainty — you can see the number, but the fading is the tool saying “don’t treat this like a full-precision measurement.” It doesn’t hide the value, because the point estimate is still your best guess. It just refuses to render it at the same visual weight as the well-sampled buckets sitting next to it.
The same logic applies whenever you look at a subset of a bigger dataset: a “last-10-releases” view of an already n=87 bucket is going to have an in_line-sized point estimate with a big_miss-sized CI. If you find yourself excited about a pattern in a short window, always check what happens to the CI.
Open the NFP EURUSD bucket table → Free forever. No signup. No email required.