House Prices: A Stacked Ensemble Across CatBoost, XGBoost, and LightGBM

Predict a home's sale price from 79 property features (Kaggle's Ames, Iowa House Prices competition) — a benchmark regression problem where the real difficulty isn't fitting any single model, it's handling a long tail of correlated, mixed-type features (dozens of quality/condition ratings, areas, and categorical construction details) without one encoding scheme handicapping one model family over another.

Whether a single boosted-tree model is sufficient, or whether combining differently-encoded models through a properly leakage-free stacked ensemble measurably improves on any one model alone.

1,460 labeled training sales and 1,459 unlabeled test sales, each with 79 property features (lot size, neighborhood, quality/condition ratings, basement/garage/porch details, sale type and condition, and more). The target, SalePrice, is log1p-transformed before training to match the competition's actual log-RMSE scoring metric.

A single cleanup step produces three parallel encodings of the same median/mode-imputed data — raw categoricals for CatBoost, label-encoding for LightGBM and XGBoost, and one-hot encoding for linear models (generated but never consumed by a model). Three base learners are trained with matched hyperparameters (depth 6, learning rate 0.03): CatBoost, XGBoost (row/column subsampling at 0.8), and LightGBM (31-leaf leaf-wise trees, same subsampling) — each with both a single full-data fit and a 5-fold cross-validated variant with early stopping after 200 rounds without improvement. Two ensembling layers sit on top: a simple unweighted average of the three base submissions, and a proper stacked generalization that collects 5-fold out-of-fold predictions from all three base learners (never letting a model predict a fold it was trained on) and fits a Ridge regression meta-learner on those three OOF prediction columns.

Each base learner (CatBoost, XGBoost, LightGBM) serves as its own baseline for the ensembling question — the relevant comparison is whether the blended or stacked submission beats the single best base model, not whether boosting beats a naive predictor.

5-fold cross-validation (plain KFold, since this is a regression problem) scored on RMSE in log-SalePrice space per fold, matching the competition's own scoring. The stacking meta-learner is evaluated via its own OOF RMSE across the same 5 folds, computed from predictions no model ever trained on.

The last logged CatBoost cross-validation fold reached log-RMSE ≈ 0.1060 at iteration 3,368 (early-stopped from a 5,000-iteration budget, patience 200) — competitive with typical solid-single-model Kaggle leaderboard scores (roughly 0.11–0.13) for this competition. All three base models and both ensemble variants (average blend, Ridge stack) produced complete, correctly-shaped 1,459-row submissions.

Only the last-run fold's training curve survives on disk — CatBoost's diagnostic directory is overwritten on every run — so the 0.106 figure is one fold's number, not a saved, verified mean-across-folds score. None of the training scripts persist their printed mean/std CV RMSE or the stacking meta-learner's OOF RMSE to a results file; they exist only in console output from whichever run happened last, so the actual relative ranking of blend vs. stack vs. best single model can't be reconstructed from the repo alone.

A one-shot competition submission pipeline — five submission CSVs (catboost, lgbm, xgb, blend, stack), no inference API, no serving path. The three parallel data-encoding branches (CatBoost, LightGBM/XGBoost, one-hot linear) is a reusable pattern worth carrying into a real project, since it avoids forcing a single encoding scheme onto model families with different native handling of categoricals.

None — static training against a fixed historical dataset, run once per script invocation with no persisted experiment tracking across runs.