Smart Score Prediction Accuracy

How to read this

Every day after a fresh NY Lottery snapshot, we re-grade the previous day's Smart Score ranking against what actually happened. The "actual" outcome per game is a weighted composite: 30% Δpayout, 20% Δexpected value, 30% prize-claim velocity, 15% pool drawdown, 5% top-prize hit. We then compute standard ranking metrics:

• Pairwise accuracy — across every pair of games, did we get the order right? 50% is random; >55% is real signal.
• Spearman ρ / Kendall τ — overall rank correlation in [-1, 1].
• NDCG@10 — discounted-cumulative-gain at top-10 (how well our top-10 ordering captures the actual top-10 by composite outcome).
• Top-5 / Top-10 precision — fraction of our top-K that landed in the actual top-K. With only ~65 games in the active set, expected random precision for Top-5 is ~7.7%, so trends matter more than absolute values.

Smart Score validation — full daily history (81 runs)

Every published validation day, no aggregation, newest first.

ML predictor (H=1) validation — full history (51 runs)

The bagged-GBM ML predictor produces a separate ranking each day. It's graded against the same outcome composite as Smart Score, so the gap between the two tells us how much lift the ML layer adds over the heuristic baseline.

ML predictor (H=7) — 7-day-horizon

A parallel model trained on the same feature set predicts smart_score 7 days out. With less day-to-day noise, the H=7 head is expected to converge to higher Top-K precision once enough graded files accumulate.

Warming up — first H=7 prediction will mature 7 days after model deploy. Check back after the first H=7 file ages out.

Training configuration (live)

• Trees: 15 (auto-tuned: bumps up if 5-day pairwise drops below 55%, trims down if it stays above 75%)
• Train window: 90 days of historical snapshots
• Learning rate: 0.06
• Cumulative training rounds: 1224
• Features: 33 Smart Score factors + edge/timing axes + price/lifecycle/trend/velocity/competition + pack-turnover (retailer demand proxy) + Cox-style top-prize hazard + linear-depletion residual + interaction terms.

Open data

This dashboard is generated server-side from accuracy/history.json (Smart Score grading), prediction_accuracy/history.json (ML H=1), and prediction_accuracy_h7/history.json (ML H=7), plus ml/rank_prediction_model.json for training stats. All files refresh every cycle. Researchers and journalists may cite the numbers under CC BY 4.0; please link back to scratchoffsny.com/methodology/performance.

• /rank-accuracy-history — Smart Score validation API (JSON)
• /prediction-accuracy-history — ML H=1 validation API
• /prediction-accuracy-history?horizon=7 — ML H=7 validation API