Linear regression — the model you must understand

📅 Day 37. Linear regression is the most underestimated model in machine learning. It's also the one you must understand before any neural net makes sense.

🧮 The whole model in one equation — y = X·w + b. Output is a linear combination of features (X·w) plus a bias (b). Learn the weights w and bias b such that the predicted y is close to the actual y on training data.

📊 'Close' usually means minimum squared error. Sum of (predicted - actual)^2 across all training examples. Take derivative; set to zero; solve. The result has a closed-form solution — the normal equations. No gradient descent needed for small problems.

🎯 Why it's foundational:

1️⃣ It's the baseline you compare every fancy model against. A model that doesn't beat linear regression on tabular data is usually not worth deploying.

2️⃣ Its assumptions — linearity, independence of errors, normal residuals — are the assumptions that deep learning violates and gets away with. Understanding what assumption is being violated tells you what model to reach for next.

3️⃣ The math (least squares, maximum likelihood under Gaussian errors) underlies almost every ML loss function you'll ever see. MSE in neural networks is least squares. Cross-entropy in classification is maximum likelihood under a different distribution. The structure repeats.

💡 If linear regression solves your problem, ship it. Less code, easier to debug, faster to retrain, more interpretable. The bias toward complex models is a self-imposed cost.

🚀 Master this before transformers. The transformer's final layer is, mathematically, a linear regression on top of learned features.

📊 Three regression metrics, three different stories about your errors. Pick by what kind of error matters most for your problem.

📐 MSE — Mean Squared Error. Sum of (y_true - y_pred)^2 divided by N. Penalises large errors quadratically. A prediction that's 10 off contributes 100 times more to the loss than one that's 1 off. Smooth gradient (good for training neural nets). Sensitive to outliers (a few bad predictions dominate the average).

📏 RMSE — Root Mean Squared Error. Square root of MSE. Same units as your target (helpful for interpretation). 'My RMSE is $1200' means the model is typically off by about $1200 on house price predictions. Inherits MSE's outlier sensitivity.

📐 MAE — Mean Absolute Error. Average of |y_true - y_pred|. Linear in error. A prediction off by 10 contributes 10x more than one off by 1, not 100x. Robust to outliers. Less smooth gradient (some optimisers prefer MSE).

🎯 The choice depends on what you're optimising for:

→ Errors should hurt proportionally → MAE.

→ Big errors are catastrophically worse than small ones (forecasting demand, where being short by 100 units costs more than being short by 10) → MSE/RMSE.

→ Median outcome matters more than mean → MAE.

💡 My rule of thumb — report BOTH RMSE and MAE in evaluation. The gap between them tells you about outlier influence. If RMSE is much larger than MAE (like 3x), outliers are dominating; investigate before trusting the model.

📋 For papers and reports — pick one as the primary metric, mention the other for context. Don't bury the inconvenient one.

💻 Two implementations of linear regression. Three lines with sklearn. Two lines with NumPy. Worth seeing both — you understand what sklearn is doing under the hood.

📦 Sklearn version. Three lines. LinearRegression() creates the model. .fit(X, y) computes the weights and bias from training data. .predict(X_new) generates predictions. Done.

🧮 Closed-form NumPy version. Two lines after the augmentation. We add a column of ones to X (so the bias gets absorbed into the weight vector). Then the formula w = (X^T X)^-1 X^T y gives us the weights directly. This is the closed-form solution to the least-squares problem.

The pseudo-inverse (np.linalg.pinv) handles the case where X^T X is singular — happens when features are perfectly collinear or when you have more features than samples. Numerically stable; standard practice.

⚖️ Why both versions exist. Sklearn's LinearRegression internally does almost exactly this — solves the normal equations with a numerically stable method (LAPACK's gelss or gelsd, depending). Wrapping it gives you scoring methods, parameter persistence, sklearn pipeline integration. The two-liner skips all that.

🚀 For small problems — closed-form is exact and fast. O(d^3) for inversion, where d is feature count. Fine for d up to a few thousand.

📈 For large problems (millions of features, millions of samples) — gradient descent is faster. SGD updates weights one batch at a time without ever materialising X^T X.

💡 Knowing both gives you the connection from classical ML to deep learning. Same loss function. Different optimisation. Same idea.

💡 Tonight's pro tip — before you report any model's performance, beat the dumb baseline. By 'dumb baseline' I mean the simplest possible thing that predicts.

🤔 What's the dumb baseline?

📊 For regression — predict the mean of the training target. Always. For every test input. Compute RMSE.

🎯 For classification — predict the majority class. For every test input. Compute accuracy / F1.

💼 The dumb baseline gives you a floor. Your model has to beat it to be worth the compute, the deployment effort, and the maintenance burden.

📈 Sklearn has DummyRegressor and DummyClassifier built in for exactly this. Three lines to set up, instant baseline.

⚠️ I've seen plenty of 'great models' that beat the dumb baseline by 0.5%. Most aren't worth deploying. The cost of running an ML model in production — inference latency, monitoring, retraining, on-call rotations for when it breaks — needs to be earned by meaningfully better predictions.

💵 Quantify the win. If the dumb baseline has 78% accuracy and your model has 79%, the marginal value of one percentage point depends on the use case. For high-stakes problems (medical diagnosis), 1% is huge. For ad clickthrough prediction at scale, 1% might be huge or might be invisible noise.

🎯 The discipline — always report the gap, not just the score. 'Our model achieves 84% accuracy, vs 76% for the majority-class baseline (+8 points).' Now the reader knows what the model actually contributes.

💡 Beating yourself isn't the goal. Beating the dumb baseline by a meaningful margin is.

📅 End of Day 37. Linear regression done.

✅ Recap:

🧮 Linear regression in one equation — y = Xw + b. The simplest model and the most underrated. Foundation under every loss function you'll see.

📊 MSE, RMSE, MAE — pick by error shape. MSE/RMSE punish large errors quadratically. MAE is robust to outliers. Report both.

💻 Sklearn (3 lines) vs closed-form NumPy (2 lines). Same answer; sklearn wraps the same math with conveniences.

🎯 Always beat the dumb baseline. DummyRegressor / DummyClassifier give you a floor. 0.5% gain over baseline rarely justifies the production cost.

🧠 A broader thought — linear regression is what you compare to when you want to know if your fancy model is worth the complexity. If LightGBM beats linear regression by 8 points, ship LightGBM. If by 0.5 points, ship linear regression — same accuracy, fraction of the maintenance.

🚀 Tomorrow, Day 38 — logistic regression. Same math; different output. The bridge to neural networks. Sigmoid squashes the linear output to a probability; cross-entropy loss replaces MSE; the model becomes a single neuron.

💼 Once you understand linear and logistic regression, you've understood 80% of the math behind every ML model — including modern deep learning. The rest is bigger and stacked.

👋 See you in the morning.