~/projects/nba-player-performance-forecasting

NBA Player Performance Forecasting: CatBoost vs. Random Forest

Team analysts, media, and award voters all want a credible answer to "how will this player perform next season" before the season happens — but naive year-over-year projections ignore usage changes, aging curves, and role shifts already latent in the box score.

Ongoing ~/projects

01// problem.md

Problem

02// decision.md

Decision Supported

Whether a forecasted stat line for a player's next season is trustworthy enough to inform roster, media, or award-conversation decisions — and which of two model families to trust for that forecast.

03// dataset.md

Dataset

Historical NBA per-game, totals, and advanced box-score statistics spanning the 1979–80 through 2023–24 seasons (45 seasons), engineered into lag and rolling-window features per player-season, with the actual 2024–25 season held out entirely for testing.

04// architecture.md

Architecture

Two independently trained multi-output models: (1) a CatBoost regressor (MultiRMSE loss, 7,000 trees) predicting 80 next-season targets simultaneously — totals, per-game, and advanced stats — from 403 numeric features plus 5 categorical features; (2) a scikit-learn MultiOutputRegressor wrapping 500-tree random forests, predicting 9 core per-game and advanced targets from roughly 100 engineered features including 1-season lags.

05// baseline.md

Baseline

A naive persistence baseline (predict a player's next season as unchanged from their most recent season). An early internal evaluation script also served as an unintentional baseline check — it measured accuracy by reconstructing per-game stats from that same season's own totals with a random-player train/test split, which turned out to be a near-tautological setup rather than a real forecast (see Limitations).

06// evaluation.md

Evaluation Framework

A true out-of-time backtest: train through 2023–24, predict 2024–25, then score against the real 2024–25 results — not a random train/test split, which for this task substantially overstated accuracy. Metrics: per-target RMSE, MAE, and R² across a held-out set of roughly 450 real players.

07// results.md

Results

On the 9 stats both models predict, CatBoost averaged R² = 0.54 versus Random Forest's R² = 0.44 in the honest next-season backtest — driven mainly by CatBoost holding up better on shooting-efficiency targets (3P%, true shooting %), where Random Forest's R² went negative (worse than just guessing the league average). On raw counting stats (points, assists, rebounds, turnovers) the two models were roughly tied.

08// error_analysis.md

Error Analysis

Retroactively ranking players by how far they beat each model's forecast reproduces a surprisingly good Most Improved Player (MIP) signal. Restricting to players with an established prior-season role (≥20 minutes/game, ≥40 games) and ranking by forecast-beating margin, Random Forest placed the actual 2024–25 MIP winner (Dyson Daniels) at #2 of 218 eligible players, with the top-4 real vote-getters all landing in the top 15; CatBoost was directionally right but noisier at the very top (rank #8). Both models completely missed on the 6th-place vote-getter, Evan Mobley — his case leaned on defensive impact and team-success narrative that box-score lag features don't capture. Rank correlation between model output and real MIP vote order: 0.53 (Random Forest) and 0.46 (CatBoost).

The real 2024–25 NBA Most Improved Player vote, next to where each model ranked that same player among the 218 players with an established 2023–24 role (≥20 minutes/game, ≥40 games) — ranked purely by how far actual 2024–25 performance beat the model's pre-season forecast, with no knowledge of the vote.

Vote Rank	Player	Vote Share	CatBoost Rank	RF Rank
1	Dyson Daniels	0.664	#8 (2.49)	#2 (2.77)
2	Ivica Zubac	0.372	#2 (3.48)	#4 (2.23)
3	Cade Cunningham	0.244	#25 (1.80)	#15 (1.49)
4	Christian Braun	0.240	#10 (2.40)	#3 (2.58)
5	Austin Reaves	0.078	#20 (1.93)	#29 (1.16)
6	Evan Mobley	0.042	#113 (0.50)	#52 (0.66)
7	Amen Thompson	0.036	#6 (2.51)	#10 (1.69)
8	Deni Avdija	0.034	#54 (1.28)	#61 (0.57)
9	Tyler Herro	0.032	#42 (1.40)	#50 (0.73)
10	Josh Giddey	0.026	#94 (0.74)	#20 (1.38)
11	Payton Pritchard	0.012	#15 (2.12)	#24 (1.28)
12	Toumani Camara	0.010	#51 (1.32)	#21 (1.36)
13	OG Anunoby	0.006	#17 (1.99)	#23 (1.32)
14	Jalen Williams	—	#52 (1.32)	#31 (1.11)

Rank correlation vs. real vote (CatBoost)

0.46

Rank correlation vs. real vote (Random Forest)

0.53

09// deployment.md

Deployment Considerations

Not deployed as a live service; runs as an offline analysis pipeline. Any production use would need to re-train each offseason, re-run the out-of-time backtest every year (accuracy on one season doesn't guarantee the next), and treat outputs as a prior to combine with qualitative scouting input rather than a stand-alone forecast.

10// monitoring.md

Monitoring Approach

No live monitoring yet — the only check so far is the season-end backtest against real outcomes described above. A production version would track rolling per-target RMSE/R² as each new season completes and flag drift if error grows relative to the historical backtest.

11// limitations.md

Limitations

An early internal evaluation script measured the Random Forest by predicting per-game stats from that same season's own totals with a random-player train/test split — a near-tautological setup that reported R² = 0.94, dramatically overstating real forecasting skill versus the true out-of-time R² of 0.44.

Both models are box-score-only: they have no way to see a trade, a coaching change, or an expanded role coming, which is exactly what tends to produce the biggest real breakouts.

The 80-target CatBoost model's individual player forecasts get noisy for aging veterans and deep bench players — one exploratory run flagged implausible late-career scoring jumps for players well past their peak, even though its aggregate accuracy across all ~450 players is solid.

"Confidence" estimates differ by model architecture (bagged-tree vote spread for Random Forest vs. training-convergence stability for CatBoost) and are not on a comparable scale to each other.