Neurons & perceptrons — DL from first principles

📅 Day 43. Week seven begins. Welcome to deep learning.

🧠 Strip away every textbook diagram you've ever seen of neural networks. The atomic unit is shockingly small.

A neuron is — output = f(W·x + b)

That's it.
→ x is the input vector.
→ W is a weight vector (one weight per input feature).
→ W·x is a dot product (linear combination).
→ b is a bias (a single scalar shift).
→ f is a non-linear activation function (ReLU, sigmoid, tanh, GELU).

A single neuron computes a weighted sum of its inputs, adds a bias, and passes the result through a non-linearity. That's the whole story.

🎯 A single neuron with sigmoid activation IS logistic regression from yesterday. Same math. Same loss function. Same training procedure. The neuron is just the building block.

🏗 Stack neurons in a layer (n neurons, each producing one output from the same input). Stack layers (the output of layer 1 is the input to layer 2). Add a final output layer with the right activation for your task (sigmoid for binary, softmax for multi-class, none for regression).

You now have a multi-layer neural network. Universal approximator — given enough neurons and the right training, it can fit any function.

🌐 That's the entire blueprint behind modern AI. GPT-4 has a few hundred billion of these neurons stacked across many layers. The architecture details (transformers, attention) are about HOW to stack and connect them efficiently. The atomic unit is still f(W·x + b).

💡 Internalise the equation. The rest is plumbing.

🧠 Activation functions are the non-linear functions f() in our neuron equation. Without them, stacked neurons would collapse into a single linear function (linear composed with linear is linear). The non-linearity is what gives neural networks their power.

📊 The six activations you'll meet:

⚡ ReLU. max(0, x). The default for hidden layers since around 2012. Fast (no exp), sparse (negative values become 0), simple gradient. Sometimes 'dies' — neurons whose output is always 0 stop training. Leaky ReLU fixes this by allowing a small negative slope.

🌊 GELU. A smoother variant of ReLU. The default in transformers and most modern architectures. Slightly more compute, slightly better behaviour. If in doubt, GELU.

🎯 Sigmoid. σ(x) = 1 / (1 + e^-x). Output bounded to (0, 1). Used in the OUTPUT layer for binary classification (gives a probability). Almost never used in hidden layers anymore (suffers from vanishing gradients on deep networks).

🎯 Softmax. Generalisation of sigmoid to multiple classes. Output is a probability distribution over k classes (sums to 1). Used in the OUTPUT layer for multi-class classification.

📐 Tanh. (e^x - e^-x) / (e^x + e^-x). Output bounded to (-1, 1). Used in RNNs and a few other places where you want zero-centered output.

🩹 Leaky ReLU. max(0.01*x, x). Same as ReLU but with a small slope for negative inputs. Use when you suspect 'dying ReLU' is happening.

📋 The rule:
→ Hidden layers — ReLU or GELU.
→ Binary output — sigmoid.
→ Multi-class output — softmax.
→ RNN hidden state — tanh.
→ ReLU dies — leaky ReLU.

💡 90% of layers are ReLU or GELU. Don't overthink the activation choice.

💻 The simplest non-trivial neural network — a 2-layer multilayer perceptron (MLP). Linear → ReLU → Linear. Trains MNIST in under a minute. Read this code; you've read 80% of every PyTorch model.

Look at the snippet.

🏗 Class MLP inherits from nn.Module. Every PyTorch model does this. nn.Module gives you parameter tracking, GPU movement, and serialisation for free.

📦 In __init__, we define the layers as instance attributes. nn.Sequential stacks layers in order. nn.Linear(in, out) is a linear transformation (the 'Wx + b' from the morning post). nn.ReLU() is the activation function as a layer (callable).

🔄 forward(x) defines the forward pass. PyTorch calls this when you do model(x). The Sequential takes care of running each layer in order.

📊 Test — model = MLP(784, 256, 10). 784 inputs (flattened 28x28 MNIST image). 256 hidden units. 10 outputs (one per digit class). Then model(torch.randn(8, 784)) runs a batch of 8 random inputs through the model. Output shape is (8, 10) — one row of 10 logits per input.

🎯 What's missing for actual classification — softmax (often baked into the loss function, like CrossEntropyLoss). Loss function. Optimiser. Training loop. We cover those tomorrow.

💡 The pattern — define layers in __init__, run them in forward. The model is a compositional structure. To add a layer, add a line. To make it a CNN, swap nn.Linear for nn.Conv2d.

🚀 12 lines. Real neural network. PyTorch's API is small enough to fit in your head; powerful enough to build any architecture you've heard of.

💡 Pro tip — bad weight initialisation can prevent training from making progress at all. The symptom — your loss flatlines from epoch 1, no matter how long you train.

🚨 The classic failure — initialise all weights to zero. Now every neuron in a layer computes the same output, gets the same gradient during backprop, updates the same way. They never differentiate. The network has the capacity of a single neuron.

✅ The fix — initialise from a small random distribution. The exact distribution depends on the activation function.

🎯 Kaiming initialisation (He init). For ReLU and its variants. Variance scaled by 2 / fan_in (number of input units). Designed to keep activation magnitudes stable across layers in deep networks.

nn.init.kaiming_normal_(layer.weight, nonlinearity='relu')
nn.init.zeros_(layer.bias)

🌊 Xavier initialisation (Glorot init). For sigmoid and tanh. Variance scaled by 2 / (fan_in + fan_out). Better for activations that have a different variance profile.

nn.init.xavier_normal_(layer.weight)

📦 Good news — PyTorch's defaults are usually fine. nn.Linear initialises with Kaiming uniform by default. nn.Conv2d does too. You only need to override for unusual architectures or when you're debugging weight scale issues.

🔍 Diagnosis — if loss flatlines:

1️⃣ Check if outputs are saturated (all close to 0 or 1 for sigmoid). Bad init or bad scaling.

2️⃣ Check gradient norms. If they're zero or NaN, the gradients aren't flowing. Init or normalisation issue.

3️⃣ Try a smaller learning rate. Sometimes init is fine but lr is too high and weights diverge in step 1.

4️⃣ Check for dying ReLU. Many neurons stuck at zero output → switch to leaky ReLU.

💡 Most of the time, defaults work. When they don't, the diagnostics above narrow it down fast.

📅 End of Day 43. Welcome to deep learning week.

✅ Recap:

🧠 Neuron = f(Wx + b). The atomic unit. Single neuron with sigmoid is logistic regression. Stack neurons; stack layers; you have a neural network.

📊 Activations — ReLU/GELU for hidden, sigmoid/softmax for output, tanh for RNNs. Don't overthink.

💻 12-line MLP in PyTorch. nn.Module class, nn.Sequential composition, forward method. The pattern repeats for every model.

🎯 Initialisation matters. Zero init kills training (symmetry trap). Kaiming for ReLU, Xavier for sigmoid/tanh. PyTorch defaults usually fine.

🧠 The bigger picture — modern AI is built from this single equation, repeated billions of times in stacked layers. Transformers add attention; CNNs add convolutions; the underlying neuron is the same. Once you grok the unit, the architectures are just different ways of connecting them.

🚀 Tomorrow, Day 44 — backpropagation. The chain rule that makes deep networks trainable. The optimiser families (Adam, AdamW, SGD). The training loop you'll write 100 times. The 'NaN loss? Lower lr first' rule.

💼 We're 48% through. Foundations + Python + DSA + data + classical ML + (now) DL fundamentals. Three more weeks of ML/DL/NLP. Then RAG, agents, automation, career.

👋 See you in the morning.