Random forests + gradient boosting — tabular kings

📅 Day 40. Tree ensembles dominate tabular ML. Two strategies for combining trees, two different ways of thinking about model errors.

🌲 Random Forest (bagging). Train many trees in parallel. Each tree sees a different bootstrap sample of the training data (sampled WITH replacement). Each tree is independently 'wrong' in different ways. Average the predictions. The variance averages out; the bias stays roughly the same.

Key hyperparameters — n_estimators (number of trees, usually 100-500), max_features (features to consider per split, typically sqrt of total). Trees can be deep; the ensemble averaging prevents overfitting.

🚀 Gradient Boosting (XGBoost, LightGBM, CatBoost). Train trees sequentially. Each new tree is trained to correct the residual errors of the ensemble so far. The model 'boosts' itself by progressively fixing what it gets wrong.

Key hyperparameters — n_estimators (usually 100-1000), learning_rate (how much to shrink each tree's contribution, usually 0.01-0.1), max_depth (typically shallower than RF, like 3-8). Trees stay small; the boosting handles complexity.

📊 Bias vs variance:

→ Random Forest reduces variance (high-variance trees averaged → stable ensemble).
→ Gradient Boosting reduces bias (each tree corrects the previous, fitting more closely to the truth).

🏆 In 2026, gradient boosting (LightGBM or XGBoost) is the strongest baseline for tabular data. It beats deep learning on most structured datasets. Kaggle competitions are won by gradient boosting more than any other technique.

💡 Random forest is more robust out of the box (less hyperparameter tuning needed). Gradient boosting needs more care but ceiling is higher. For production tabular ML — start with LightGBM defaults, tune learning_rate and n_estimators with early stopping.

🌲 In the gradient boosting world, three libraries dominate — LightGBM, XGBoost, CatBoost. All are excellent. In practice, I reach for LightGBM first.

⚡ Why LightGBM:

→ Faster training. Especially on large datasets. Histogram-based splits are 5-20x faster than the exact-split methods.

→ Native categorical feature support. Pass categorical_features=[...] and LightGBM handles them without one-hot encoding. Faster, less memory, often better accuracy.

→ Lower memory footprint. Important when you're working with millions of rows.

→ Cleaner Python API. lgb.LGBMClassifier follows sklearn conventions tightly.

→ Excellent default values. The out-of-box performance is usually within 1-2% of well-tuned XGBoost.

🦾 Where XGBoost still wins:

→ Stability. Older, more battle-tested. Bug rate has been near zero for years.

→ Sklearn-style API consistency. Some teams use it everywhere for uniformity with scikit-learn pipelines.

→ Some specific algorithms it has that LightGBM doesn't (DART, custom objective functions in slightly different shapes).

🐱 Where CatBoost wins:

→ Strongest categorical handling out of the box. Uses target statistics with regularisation.

→ Best defaults of the three for most problems. Slightly slower to train.

→ Built-in handling of overfitting via 'ordered boosting'.

📋 My decision tree:

→ Mostly numerical features → LightGBM.
→ Lots of categorical features → CatBoost or LightGBM with categorical_features=[].
→ Already have an XGBoost-based codebase → stick with XGBoost.
→ Maximum performance ceiling, willing to tune extensively → all three with grid search; pick winner.

💡 Try LightGBM first. Switch only if you hit a specific issue.

💻 Most LightGBM examples online have 30 hyperparameters set explicitly. You don't need them. Start with this 10-line setup and a single early-stopping callback.

Look at the snippet. We create LGBMClassifier with four parameters:

→ n_estimators=500. The maximum number of trees. With early stopping, the actual number is whatever stops improving.

→ learning_rate=0.05. The shrinkage factor for each tree. Lower = more trees needed but smoother fit. 0.05 is a good middle ground.

→ num_leaves=63. Maximum leaves per tree. LightGBM uses leaf-wise (best-first) growth, so this controls tree complexity. 63 is the default and works well.

→ random_state=42. Reproducibility.

🎯 The fit call passes:

→ Training data.

→ eval_set with validation data. LightGBM scores on this set during training.

→ callbacks=[lgb.early_stopping(20)]. Stop training if validation score doesn't improve for 20 rounds. Picks the best iteration automatically.

📈 Why early stopping is non-negotiable. Without it, you'd train for the full n_estimators (500) and likely overfit. With it, training stops as soon as validation accuracy peaks. Often the actual best iteration is around 100-300, not 500.

🏆 The output AUC tells you how well the model separates classes. Above 0.85 is decent for many real-world problems. Above 0.95 is suspicious — check for leakage.

💡 The 30-line LightGBM examples online often tune subsample, colsample_bytree, lambda_l1, lambda_l2, min_child_samples, etc. Start without these. After you have a baseline working, do hyperparameter tuning (Optuna is great for this). The marginal improvement from extensive tuning is usually 1-3%; the time cost is huge.

💡 Tonight's pro tip — every tree-based model gives you feature_importances_ for free. Always look at them. They reveal more about your data than the model's accuracy.

📊 What it shows. A score per feature reflecting how much each contributed to reducing impurity across all trees. Higher = more important. Sort and plot the top 20 as a bar chart.

🔍 What you'll learn:

→ Which features actually drive the model. Often surprising. The feature you thought was critical might be ranked 12th; the one you almost dropped might be #1.

→ Suspected leakage. If 'created_at' or 'customer_id' or some derived field is at the top, investigate. Is this feature genuinely predictive or is it secretly the answer?

→ Feature engineering opportunities. If a date field is high but you're using it as a string, parse it into year/month/dayofweek/quarter and likely improve the model.

→ What to drop. Features at the bottom (importance near zero) can be removed for simplicity without much accuracy loss.

⚠️ Caveat — built-in importance can be biased toward high-cardinality features. A categorical with 1000 levels might rank high simply because there are many split opportunities, not because it's truly informative.

📈 The honest version — use SHAP values (shap library). SHAP gives:

→ Per-prediction attribution. 'For this customer, age contributed +0.3 to the prediction.'

→ Global importance that's bias-corrected.

→ Direction of effect (does high feature value increase or decrease the prediction?).

SHAP runs slower than feature_importances_ but the insights are sharper.

💡 The best ML interview answer is — 'I plotted feature importance and noticed that timestamp_created was at the top, which turned out to be a leaky timestamp from the future.' That kind of observation is what separates ML engineers who ship from those who don't.

📅 End of Day 40. Five-eighths through week six.

✅ Recap:

🌲 Bagging vs boosting in two paragraphs. RF averages parallel trees (variance reduction). GBM trains sequential trees that fix errors (bias reduction).

⚡ LightGBM > XGBoost in most cases. Faster, native categoricals, lower memory, cleaner API. Try LightGBM first.

💻 10-line LightGBM with early stopping. Forget the 30-parameter tutorial. Start simple, tune later.

📊 feature_importances_ — the cheapest insight in tree-based ML. Catches leakage, reveals what matters, suggests feature engineering.

🧠 The honest reality of tabular ML in 2026 — gradient boosting wins. Most production systems handling structured data (fraud, churn, ranking, demand forecasting) use LightGBM or XGBoost. Deep learning (TabNet, FT-Transformer) is catching up but hasn't decisively won. For most tabular problems, GBM is the right default.

🚀 Tomorrow, Day 41 — clustering. K-means, when there are no labels. The elbow method and silhouette score for picking k. Why you ALWAYS scale before clustering. The cases where K-means fails (and what to use instead).

💼 We've covered linear regression, logistic regression, decision trees, ensembles. Tomorrow's clustering is a shift to unsupervised. Day 42 wraps with SVMs and the classical-vs-DL pick-by-data decision tree.

👋 See you in the morning.