Decision trees — the interpretable workhorse

📅 Day 39. Decision trees are the most interpretable model in ML — and they're also the building blocks of random forests and gradient boosting, which dominate tabular data.

🌳 The model — a flowchart. Each internal node tests one feature against a threshold ('age > 35?'). Branches lead to subtrees based on the answer. Each leaf has a prediction (a class for classification; a number for regression).

🎯 Training — pick the split (feature + threshold) that reduces 'impurity' the most. Recurse on each child. Stop when the tree is deep enough or splits don't help. Greedy, top-down. O(n × d × log n) or so.

📐 Impurity for classification — Gini or entropy. Both measure how mixed the classes are at a node. Pure node (all one class) → impurity 0. Mixed node (50/50) → impurity max. The split that maximises 'parent impurity minus weighted child impurities' is chosen.

📐 Impurity for regression — mean squared error. The split that minimises sum-of-squared-residuals across the children.

🎯 Inference — start at the root, follow branches based on the input's features, return the leaf's prediction. O(depth) time. For balanced trees, that's O(log n).

💎 Why people love decision trees:

→ Interpretable. You can literally print the tree and read it. Every prediction has a traceable path.

→ No scaling needed. Trees handle raw features fine. No StandardScaler. Categorical features (after encoding) work natively.

→ Capture non-linear interactions for free. The tree splits on whichever feature reduces impurity most; if interactions matter, splits naturally exploit them.

⚠️ Why alone they overfit — a deep tree memorises training data. That's why we ensemble them (tomorrow).

💡 Master the building block. Then we stack.

🧮 If you've read about decision trees, you've seen the Gini-vs-entropy debate. Here's the honest take — they're nearly equivalent and the debate is mostly aesthetic.

📊 Gini impurity. 1 - sum(p_i^2) where p_i is the proportion of class i at the node. Pure node → 0. Maximum mixed → 0.5 for binary. Faster to compute (no log). Sklearn's default for classification trees.

📊 Entropy. -sum(p_i * log(p_i)). From information theory. Pure node → 0. Maximum mixed → 1 for binary (when using log base 2). Slightly slower (logs are expensive). Used in some classical ML literature.

📈 Information gain — the entropy reduction after a split. parent_entropy - sum(weight_i * child_entropy_i). The split with highest info gain is chosen.

🎯 Both Gini and entropy:

→ Are zero when a node is pure.
→ Maximise when classes are perfectly mixed.
→ Decrease monotonically as the node becomes purer.
→ Produce nearly identical trees in practice. Studies show <2% accuracy difference on standard benchmarks.

💡 What MATTERS far more than Gini-vs-entropy:

1️⃣ max_depth. Caps tree depth. The single biggest hyperparameter. Default is None (grow until pure), which overfits.

2️⃣ min_samples_leaf. Minimum rows per leaf. Prevents tiny leaves that memorise training noise.

3️⃣ max_features. How many features to consider per split. Used in random forests for variance reduction.

📋 The reality — pick Gini (sklearn default), tune max_depth and min_samples_leaf, move on. Stop arguing about impurity criteria; you're not going to find the breakthrough there.

🚀 The breakthroughs come from ensembling (tomorrow), not from impurity tweaks.

💻 Sklearn's plot_tree gives you a printable tree diagram. For decision trees up to depth 5 or so, this is incredibly useful — you can literally see the splits.

Look at the snippet. We train a DecisionTreeClassifier with max_depth=4 (so the visualisation isn't too big). Then plot_tree renders the full tree with feature names, class names, and color-filled nodes (color intensity reflects class purity).

🎨 The output shows, for each node:

→ The split — feature and threshold.
→ Gini (or entropy) value.
→ Number of samples reaching this node.
→ Class distribution.
→ Predicted class.

🔍 You can trace the model's logic for any specific input by walking the tree by hand. 'Age <= 30 → goes left → Income > 50K → goes right → predicts class 1'. That's interpretation in the most concrete sense.

🚀 Use cases for tree visualisation:

→ Explaining the model to non-technical stakeholders. 'Here's exactly what it checks.'
→ Debugging suspected bugs. If the tree splits on a feature that shouldn't matter, you might have leakage.
→ Sanity-checking feature engineering. The features at the top of the tree are the most informative.

⚠️ Caveats:

→ Beyond depth 5-6, the visualisation becomes unreadable. Too many nodes.
→ For very wide datasets, individual splits might use one column out of thousands. Hard to see big picture from one tree.
→ Random forests and gradient boosting make many trees. You'd visualise one or two for intuition; the ensemble's behaviour isn't from any single tree.

💡 For deeper trees or ensembles, switch to feature_importances_ (a bar chart of feature relevance) or SHAP for per-prediction attribution.

💡 Two hyperparameters tame decision trees. Get them right and you have a useful interpretable model. Get them wrong and you have a wildly overfit tree that memorises training data.

🌳 max_depth. The deepest a tree can grow. Default is None (grow until pure or hits min_samples_leaf). Default = overfit. Always set this explicitly.

📊 Sensible ranges by dataset size:
→ Small (n < 1k): max_depth 3-5. Larger trees memorise.
→ Medium (n = 1k-100k): max_depth 5-10.
→ Large (n > 100k): max_depth 6-12. Past 12, you're usually overfitting unless features are very rich.

For random forest, deeper trees are fine because the ensemble averages out the overfit. Single trees should stay shallow.

🍃 min_samples_leaf. Minimum number of training samples in any leaf node. The tree won't split a node further if doing so would create a child with fewer than this number of samples.

📊 Sensible ranges:
→ Default 1 — overfits. Each leaf can be one row.
→ 5-50 for medium data. Forces the tree to find splits that affect meaningfully-sized groups.
→ 100+ for very large datasets where you want robust splits.

🎯 The grid search ritual:

for max_depth in [3, 5, 7, 10, 15]:
    for min_samples_leaf in [1, 5, 10, 50]:
        # train + cross-validate
        ...

Twenty model fits, validation score for each, pick the best.

📈 In sklearn, GridSearchCV does this automatically. Pass the parameter grid; it returns the best combination based on cross-validation score.

💡 The cheapest win in tree-based ML — tuning these two hyperparameters. Don't accept defaults.

📅 End of Day 39.

✅ Recap:

🌳 Tree = flowchart trained on impurity reduction. Inference is O(log n). Interpretable. Handles non-linear interactions for free.

🧮 Gini vs entropy — nearly equivalent. Pick Gini (sklearn default). Move on. Don't get stuck on the choice.

💻 plot_tree visualises trees up to depth 5-6. Useful for explaining models, debugging suspected bugs, sanity-checking features.

🎯 max_depth + min_samples_leaf are the two hyperparameters that matter most. Defaults overfit. Grid search both with cross-validation.

🧠 Reflection — single decision trees are interpretable but not super accurate. They overfit easily and have high variance (small data changes lead to very different trees). Tomorrow we ensemble them, and they become the strongest tabular ML technique most production systems use.

🚀 Tomorrow, Day 40 — random forests + gradient boosting. Bagging vs boosting. LightGBM as the modern default. The 'feature_importances_' diagnostic that catches leakage. Why these models still beat deep learning on tabular data.

💼 Tabular data is most of business ML. Knowing decision trees → ensemble → boosting is the spine of what production ML looks like in fintech, retail, ad tech, healthcare.

👋 See you in the morning.