SVMs + week 6 wrap

📅 Day 42. Last day of week six.

📐 Support Vector Machines were the dominant ML algorithm of the 2000s. They've lost ground to gradient boosting on tabular and to deep learning on unstructured data. Still worth understanding — the margin idea shows up in modern contrastive learning.

🎯 The core idea — find the hyperplane that separates two classes with the largest possible 'margin' (gap between classes). The points closest to this hyperplane are 'support vectors' (hence the name). The hyperplane is positioned to maximise its distance from these support vectors.

📊 Why max-margin matters. A separating hyperplane that's right at the boundary of one class has zero margin — small perturbations in test data cross it, leading to wrong predictions. A hyperplane that's centered between classes has high margin — robust to perturbations, generalises better.

🧮 Mathematical guarantee. Statistical learning theory provides bounds on test error based on margin. Larger margin → tighter bound → better generalisation. This was a big deal in the 2000s before neural networks empirically beat the bounds.

🌀 Kernels. SVMs can implicitly map features into much higher-dimensional space without computing the mapping explicitly. RBF (Gaussian) kernel handles non-linear boundaries. Polynomial kernel captures polynomial relationships. The 'kernel trick' was groundbreaking — non-linearity at modest computational cost.

⚠️ Why SVMs lost ground:

→ Doesn't scale to huge datasets. Training is O(n²) or worse.
→ Tabular data is owned by gradient boosting (smaller, faster, often more accurate).
→ Unstructured data is owned by deep learning.
→ Hyperparameter tuning is harder than for tree models.

💡 Where SVMs still shine — small datasets (a few hundred samples) where you need a strong baseline. Text classification with TF-IDF features used to be SVM territory before transformers. Some niche cases in finance and biology.

🚀 Worth knowing for the principles. Margin, kernels, support vectors — all concepts you'll see in modern self-supervised learning.

🤔 The eternal question — should I use classical ML or deep learning? My decision tree based on data type and size.

📊 Tabular data, small (<10M rows):

→ LightGBM / XGBoost. Often beats neural networks. Fewer hyperparameters to tune. Faster to train. More interpretable. Production-friendly.

📈 Tabular data, large (>100M rows):

→ Still gradient boosting in most cases. Modern variants handle large data well.
→ Consider TabNet, FT-Transformer, SAINT for very large tabular. They sometimes win when you have the compute and data.

🖼️ Images, audio, video:

→ Deep learning, no contest. CNNs (ResNet, EfficientNet) for images. Vision Transformers (ViT) for the highest accuracy on big data. Wav2Vec for audio.

📝 Text:

→ Small data (< 10k labeled): TF-IDF + classical model often wins. Less overfitting risk.
→ Medium data: BERT-style encoders or sentence-transformers for embedding + classical model.
→ Large data: Fine-tune a transformer or use a pretrained one with a small classification head.

📅 Time series:

→ Start with classical. ARIMA, Prophet (Facebook's), exponential smoothing. Strong baselines.
→ Upgrade to neural (LSTM, Transformer-based) only if classical doesn't meet requirements and you have lots of training data.

🌐 Graph data:

→ Neural (Graph Neural Networks) for prediction tasks on graphs.
→ Classical algorithms (PageRank, community detection) for non-prediction tasks.

💡 The meta-rule — don't reach for deep learning by reflex. Many problems don't have enough data for neural networks to outperform simpler models. Many production teams underestimate the maintenance cost of DL pipelines.

🎯 Default decision tree:

1️⃣ Try a baseline (linear, logistic).
2️⃣ Try gradient boosting.
3️⃣ If you need more, then deep learning.

Most projects stop at step 2 and ship. That's fine.

💻 Sklearn Pipelines wrap preprocessing + model into a single fit/predict object. Train it; ship it; never worry about preprocessing drift between training and production.

Look at the snippet. Three steps:

1️⃣ SimpleImputer(strategy='median') — fills missing values with the median of each column. The fitted imputer remembers which median to use for each column.

2️⃣ StandardScaler() — scales features to mean=0, std=1. The fitted scaler remembers each column's mean and std.

3️⃣ LogisticRegression(max_iter=2000) — the actual model.

Wrap all three in a Pipeline. Now pipe.fit(X_train, y_train) trains the imputer, then the scaler, then the model — in order, automatically. pipe.predict(X_new) does the same chain at inference: impute, scale, predict.

🎯 Why this matters in production:

→ Single object to serialise. joblib.dump(pipe, 'model.pkl') saves the entire chain. No 'where's my scaler' debugging.

→ No preprocessing drift. Training and inference use the SAME scaler with the SAME mean/std. Common bug — train with one scaler, infer with a different one (or none) — eliminated.

→ Cross-validation done right. Without a pipeline, you might fit the scaler on the full training data, then split for CV — leaking data from validation into training. With a pipeline, the scaler is refit on each fold, no leakage.

→ Easier to swap models. Replace LogisticRegression with RandomForestClassifier; same pipeline contract.

📋 ColumnTransformer extends this for heterogeneous data — different preprocessing for different columns. One-hot encode categoricals, scale numerics, all in one Pipeline. Clean and reproducible.

💡 If your sklearn code DOESN'T use Pipeline, refactor. The 5-minute investment prevents a class of production bugs.

💡 Tonight's pro tip — when persisting sklearn pipelines, use joblib instead of pickle.

📦 Both work for sklearn objects. Both serialise the entire pipeline including fitted estimators. The difference is in efficiency and edge cases.

⚡ joblib is faster for objects with large NumPy arrays (which most ML pipelines have — fitted weights, transformer parameters, etc). It uses memory-mapped storage for arrays. The size of the saved file is similar; the load time is dramatically better for big models.

🛠 Recommended:

import joblib

joblib.dump(pipeline, 'model.pkl')
pipe = joblib.load('model.pkl')

⚠️ Important caveats for production:

1️⃣ Pin sklearn version. A pipeline trained on sklearn 1.4 may not load on sklearn 1.6. Some transformers change internal structure between versions. Pin sklearn version in requirements.txt and document it alongside the saved model.

2️⃣ Pin Python version. Pickle/joblib are sensitive to Python version differences. Train on Python 3.11, deploy on Python 3.11.

3️⃣ Pin numpy and other dependencies. Same logic.

4️⃣ Save metadata next to the model. A small JSON with the version info, training date, evaluation metrics, training data hash. Future-you will thank present-you.

5️⃣ Test the load before shipping. Save the model. Then load it from a fresh process and run inference on a known input. Verify the output matches what training produced. Catches any subtle serialisation issue immediately.

📋 For long-term storage (>1 year), consider exporting to a portable format like ONNX. Pickled models are fragile across major version jumps; ONNX is portable across runtimes.

💡 The pickle-vs-joblib choice is small. The version-pinning discipline is huge. Don't skip step 5 (test load).

📅 End of Day 42. End of week 6. We're 47% through the sprint.

✅ Week 6 in seven topics:

🎯 ML problem framing — three boxes (supervised, unsupervised, RL), the 6-step ML loop, three-way splits, picking the metric BEFORE training.

📈 Linear regression — y = Xw + b, MSE/RMSE/MAE choice, sklearn vs closed-form NumPy, always beat the dumb baseline.

🎯 Logistic regression — linear regression + sigmoid, class imbalance fixes, full diagnostics (classification_report, confusion_matrix, AUC), threshold ≠ 0.5.

🌳 Decision trees — flowchart trained on impurity, Gini = entropy, plot_tree for visualisation, max_depth + min_samples_leaf as the key hyperparameters.

🚀 Ensembles — bagging vs boosting, LightGBM > XGBoost in most cases, 10-line LightGBM with early stopping, feature_importances_ as the cheapest insight.

🎯 K-means clustering — 4-step algorithm, elbow + silhouette for picking k, StandardScaler before any distance-based algo.

📐 SVMs + classical-vs-DL decision — margin idea still relevant, gradient boosting wins on tabular, deep learning wins on unstructured. Pipeline + joblib for production.

🧠 The big takeaway from week 6 — don't reach for deep learning by reflex. Classical ML solves most production problems faster, cheaper, and with less maintenance. Save deep learning for when you genuinely need it (images, text, audio at scale).

🚀 Next week — deep learning fundamentals. Neurons, backprop, PyTorch basics, CNNs, RNNs, regularisation, training tricks. The bridge from classical ML to modern AI.

💼 47% done. The ML/AI core stack starts tomorrow.

👋 See you in week 7.