Logistic regression — classification, framed simply

📅 Day 38. Logistic regression is one of the most important models in ML — and it's literally linear regression with one extra step.

🧮 Linear regression — y = Xw + b. Output is unbounded; could be any real number.

🎯 Logistic regression — σ(Xw + b). Same Xw + b, then squashed through the sigmoid function σ(z) = 1 / (1 + e^-z). Output is now bounded between 0 and 1, interpretable as a probability.

📊 The sigmoid maps:
→ -∞ → 0
→ 0 → 0.5
→ +∞ → 1

🎯 Decision rule — if σ(Xw + b) > 0.5, predict class 1; otherwise class 0. Threshold tunable based on the cost of false positives vs false negatives.

📐 Loss function — cross-entropy (log-loss). Mathematically derived from maximum likelihood under a Bernoulli (yes/no) distribution. Computationally similar to MSE but designed for probabilities.

🧠 The bigger insight — logistic regression is a single-neuron neural network. Literally. One linear layer + sigmoid activation. Modern neural networks just stack many of these (with non-linearities between).

🚀 Why this matters for understanding deep learning. The classification head of every neural network is essentially logistic regression on top of learned features. The transformer's final layer that predicts the next token? Logistic regression on the embedding. The image classifier's final layer? Logistic regression on the convolution output. Same math; different feature pipeline.

💡 Master logistic regression and you've understood the output layer of every classifier ever built. The rest is feature engineering and stacked layers.

⚠️ Class imbalance is one of the top reasons ML models silently underperform in production.

🤔 The setup — 95% of your training rows are class 0, 5% are class 1. You train a logistic regression. Accuracy on validation is 95.2%. Looks great. Ship it.

🚨 Reality — your model probably predicts class 0 for almost everything. Accuracy is 95% because the prior alone gives you 95%. The model adds essentially zero signal on the minority class. Useless for what you actually wanted.

🎯 Four fixes for class imbalance:

1️⃣ Use a different metric. Accuracy is misleading on imbalanced data. Use F1 (harmonic mean of precision and recall on the positive class), PR-AUC (area under the precision-recall curve), balanced accuracy (mean of recall on each class). All three reward catching the minority class properly.

2️⃣ Stratify your splits. stratify=y in train_test_split. Preserves class proportions in each split. Without stratify, you might end up with 0% of the minority class in the test set on small datasets.

3️⃣ Class weights. class_weight='balanced' in sklearn auto-weights inversely by class frequency. The model now pays more attention to minority-class errors. Same data, different loss weights.

4️⃣ Resample. SMOTE creates synthetic minority examples by interpolating between real ones. Use carefully — SMOTE can create unrealistic synthetic examples in high-dim feature space. Apply BEFORE the train/test split is fine; AFTER is leakage.

📊 The first thing I do on any classification task — print value_counts() of y. If imbalanced (say >70/30), change strategy before training. Otherwise you're optimising for the wrong thing.

💻 Beyond accuracy — three diagnostics that tell the actual story of a classifier. classification_report, confusion_matrix, roc_auc_score.

📊 classification_report shows precision, recall, F1 per class — plus the overall accuracy and macro/weighted averages. One function call. Reveals whether your model is good at one class but weak at another.

📊 confusion_matrix is the 2x2 (or NxN) matrix of true vs predicted. The diagonal is correct predictions; off-diagonal is errors. Look at off-diagonal cells to see WHICH class your model confuses with WHICH other class. For a fraud model, you might see 'we miss most fraud cases' (low recall) or 'we flag too many legit transactions' (low precision). The matrix shows you exactly where.

📈 roc_auc_score — area under the ROC curve. Threshold-independent measure of how well the model separates classes. AUC of 0.5 is random; 1.0 is perfect. Useful when you want to evaluate the model's ranking ability without committing to a specific threshold.

Look at the snippet. We use class_weight='balanced' to handle imbalance, train, predict probabilities (predict_proba returns the full P(y=1|X), not just the binary class). Then we report all three diagnostics.

💡 max_iter=2000. The default in some sklearn versions is 100, which can fail to converge on harder problems. Bumping to 2000 gives the optimiser room. If it still doesn't converge, your features probably need scaling (StandardScaler) before fitting.

🎯 The lesson — accuracy alone is misleading. Always look at the full picture. precision, recall, F1, confusion matrix, AUC. Five numbers tell more than one.

💡 Pro tip that escapes most beginners — the default classification threshold is 0.5, and it's almost never optimal for your problem.

🤔 Recap — logistic regression outputs a probability. To turn it into a class prediction, you compare the probability to a threshold. By default, > 0.5 means class 1. But why 0.5?

💵 The 'right' threshold depends on the cost asymmetry of false positives vs false negatives.

🚨 Fraud detection. Missing fraud (false negative) costs the company $X per case. Flagging a legit transaction as fraud (false positive) costs Y per case in customer experience. If X > 10*Y, you should lower the threshold — call more things fraud, even if more of those are wrong, because catching the real cases matters more.

🏥 Medical screening. Missing a disease (false negative) is much worse than a false positive that leads to a follow-up test. Lower threshold; higher recall.

🛒 Spam detection. Marking real email as spam (false positive) might be much worse than letting some spam through (false negative). Higher threshold.

📈 The technique — use precision_recall_curve from sklearn. Computes precision and recall at every possible threshold. Plot them. Pick the threshold that maximises the metric you actually care about (F1, F2, F0.5 — F-beta scores weight recall more or less than precision).

🎯 In production, threshold and model are separate concerns. Train the model. Then sweep threshold against held-out data and your business metric. Deploy both together.

💡 The model gives you a probability. The threshold is your business decision. Don't conflate them.

📅 End of Day 38.

✅ Recap:

🎯 Logistic regression = linear regression + sigmoid. Output bounded to (0,1) — a probability. Single-neuron neural network in disguise.

⚠️ Class imbalance silently kills metrics. Four fixes — better metric (F1, PR-AUC), stratified splits, class weights, SMOTE. Always check value_counts(y) first.

📊 Diagnostics — classification_report, confusion_matrix, AUC. Five numbers tell more than one.

💡 Threshold ≠ 0.5. Tune by business cost. Use precision_recall_curve to pick.

🧠 The bigger picture — logistic regression is the output head of every classifier you'll ever build. Modern neural networks add layers of feature learning before this final logistic step. Understanding the math here means understanding the output of every classifier — from spam filters to image classifiers to language models predicting the next token.

🚀 Tomorrow, Day 39 — decision trees. The most interpretable model in ML and the building block of random forests + gradient boosting. The Gini-vs-entropy debate (spoiler: it doesn't matter much). The two hyperparameters that prevent overfitting.

💼 We're ramping up. Linear, logistic, trees, ensembles, k-means, SVM. By Friday we'll have covered 90% of the classical ML toolkit.

👋 See you in the morning.