Neural Network Overview
"Neural Network" is composed of two words: neuron, the fundamental building block, and network, the connections between neurons. Every neuron in a neural network holds a value, and based on that value, the neuron gets activated.
A neural network is, mechanically, nothing more than a chain of matrix multiplications and elementwise nonlinearities that gets tuned by gradient descent until its output matches the data. In this post I go through the complete theoretical training pipeline, step by step, from raw dataset to updated weights, deriving every equation involved: forward propagation, loss, backpropagation, and the weight update.
- Dataset preparation
- Initialization — weight matrices, epochs, learning rate
- Activation function definitions
- Forward propagation
- Loss computation
- Backpropagation
- Complete the epoch — new weights & biases become the model
1. Dataset Preparation
Start with a dataset of $N$ input-output pairs:
$$ \mathcal{D} = \{ (\mathbf{x}^{i}, \mathbf{y}^{i}) \}_{i=1}^{N}$$
where $x^i$ represents a single input example and $y^i$ represents its corresponding output, together forming the dataset $\mathcal{D}$ used to train the neural network.
Each input $x^i$ is represented as a column matrix of dimension $n \times 1$, where $n$ is the number of features in the model.
2. Initialization — Weights, Biases, Epochs, Learning Rate
For a network with layers $l = 1, \dots, L$, layer $l$ has a weight matrix $\mathbf{W}^{[l]}$ and a bias vector $\mathbf{b}^{[l]}$. The weight matrix $\mathbf{W}^{[l]}$ has dimension $(n^l \times n^{l-1})$, where $n^l$ is the number of neurons in layer $l$ and $n^{l-1}$ is the number of neurons in layer $l-1$. The bias vector $\mathbf{b}^{[l]}$ is a $(n^l \times 1)$ matrix. Together, this weight matrix and bias vector connect layer $l-1$ to layer $l$.
These are initialized randomly — never with all zeros, since that would make every neuron in a layer compute an identical gradient and update identically (a symmetry the network could never break).
Two more hyperparameters are fixed before training starts:
| Hyperparameter | Symbol | Role |
|---|---|---|
| Epochs (pronounced "eh-pox") | $E$ | Number of full passes over the dataset |
| Learning rate | $\eta$ | Step size for each gradient descent update |
I show how these two are actually used in Sections 6 and 7.
3. Activation Functions
If every neuron contributed equally at all times, the output would always be the same. That's why we need an activation function — it decides whether a given neuron should be activated or not. There are actually thousands of activation functions available today, and we choose one based on our use case. The most common ones are Sigmoid, Tanh, ReLU, Softmax, etc.
The derivative of each activation is what actually gets used later during backpropagation — it's the local "sensitivity" term in the chain rule. I'll describe each activation function in detail later, at the point where I actually use it.
4. Forward Propagation
Given an input vector $x$, the first layer is denoted $\mathbf{a}^{[0]} = \mathbf{x}$. Let $\mathbf{W}^{[1]}$ be the weight matrix between the input layer and layer 1, and $\mathbf{b}^{[1]}$ be the bias vector of layer 1. Then,
$$ \mathbf{z}^{[1]} = \mathbf{W}^{[1]} \mathbf{a}^{[0]} + \mathbf{b}^{[1]} $$ Passing this $z^{[1]}$ into the activation function gives $a^{[1]}$: $$ \mathbf{a}^{[1]} = f\!\left(\mathbf{z}^{[1]}\right) $$ These are the values that the neurons of layer 1 will hold.
This repeats for $l = 1, 2, \dots, L$, and the final layer's activation is the network's prediction, $\hat{\mathbf{y}} = \mathbf{a}^{[L]}$. Every $\mathbf{z}^{[l]}$ and $\mathbf{a}^{[l]}$ is cached during this pass — they're needed again during backpropagation.
So, the general equation is: $$ \mathbf{z}^{[l]} = \mathbf{W}^{[l]} \mathbf{a}^{[l-1]} + \mathbf{b}^{[l]} $$
$$ \mathbf{a}^{[l]} = f^{[l]}\!\left(\mathbf{z}^{[l]}\right) $$
5. Loss Computation
The loss $L$ measures how far the prediction $\hat{\mathbf{y}} = \mathbf{a}^{[L]}$ is from the target $\mathbf{y}$, which must be present in the dataset for training. Now you need an error function — you can choose any function you want, but for regression, mean squared error is standard:
$$ L_{\text{MSE}} = \frac{1}{n_L}\sum_{k=1}^{n_L} \left( \hat{y}_k - y_k \right)^2 $$
Over a mini-batch of size $m$, the reported loss is simply the average over examples:
$$ J = \frac{1}{m}\sum_{i=1}^{m} L^{(i)} $$
Loss ($\mathcal{L}$): The error for a single training example. Cost ($J$): The average error over an entire batch or the entire dataset.
Our job is to minimize $J$ as much as possible. When $J$ goes down, it means the network's average predictions are getting closer to the actual targets ($y$). $J$ is the scalar objective that gradient descent will minimize by adjusting every $\mathbf{W}^{[l]}, \mathbf{b}^{[l]}$ in the network.
6. Backpropagation
Backpropagation is the process of computing how much each weight and bias contributed to the final prediction error. It repeatedly applies the chain rule of calculus, propagating the error from the output layer back toward the input layer.
During the forward pass, the network stores two quantities for every layer \(l\):
$$ z^{[l]} = W^{[l]}a^{[l-1]} + b^{[l]} $$the weighted input, and
$$ a^{[l]} = f\!\left(z^{[l]}\right) $$the activation (output) of the layer. These cached values are reused during backpropagation.
Here we compute how much error is caused by a specific neuron. Let's say I want to compute the error contribution of the $a^{[1]}_4$ node. As you can see in the figure, this node is connected to nodes $a^{[0]}_1$, $a^{[0]}_2$, $a^{[0]}_3$ with corresponding weights, and it also has its own bias, which contributes to the error as well.
The Gradients (The Main Terms)
For the bias ($b_4^{[1]}$):
$$\frac{\partial C}{\partial b_4^{[1]}} = \frac{\partial C}{\partial z_4^{[1]}}$$For the weights (from any input neuron $a_j^{[0]}$ where $j \in \{1, 2, 3\}$):
$$\frac{\partial C}{\partial w_{4,j}^{[1]}} = \frac{\partial C}{\partial z_4^{[1]}} \cdot a_j^{[0]}$$7. Completing an Epoch — the Weight Update
The parameter update formulas, using your learning rate ($\eta$):
Bias update:
$$b_4^{[1]\text{(new)}} = b_4^{[1]\text{(old)}} - \eta \cdot \frac{\partial C}{\partial z_4^{[1]}}$$Weight updates:
$$w_{4,1}^{[1]\text{(new)}} = w_{4,1}^{[1]\text{(old)}} - \eta \cdot \frac{\partial C}{\partial z_4^{[1]}} \cdot a_1^{[0]}$$ $$w_{4,2}^{[1]\text{(new)}} = w_{4,2}^{[1]\text{(old)}} - \eta \cdot \frac{\partial C}{\partial z_4^{[1]}} \cdot a_2^{[0]}$$ $$w_{4,3}^{[1]\text{(new)}} = w_{4,3}^{[1]\text{(old)}} - \eta \cdot \frac{\partial C}{\partial z_4^{[1]}} \cdot a_3^{[0]}$$With every gradient computed, each parameter takes one step of gradient descent, moving opposite the gradient direction by a step size $\eta$:
$$ \mathbf{W}^{[l]} \leftarrow \mathbf{W}^{[l]} - \eta \, \frac{\partial J}{\partial \mathbf{W}^{[l]}}, \qquad \mathbf{b}^{[l]} \leftarrow \mathbf{b}^{[l]} - \eta \, \frac{\partial J}{\partial \mathbf{b}^{[l]}} $$
This is repeated for every mini-batch until the whole dataset has been seen once — one epoch — and then repeated for $E$ epochs, with the loss $J$ expected to decrease over time. There is no separate "model" object distinct from this process: the final set of weights and biases after training is the model. Everything the network "knows" is encoded entirely in the numbers inside $\{\mathbf{W}^{[l]}, \mathbf{b}^{[l]}\}_{l=1}^{L}$.
Putting It All Together
This loop is the theoretical skeleton behind essentially every neural network trained today — convolutional layers, attention layers, and normalization layers all plug into the same forward → loss → backward → update cycle, just with more elaborate versions of Sections 3 through 6. Understanding this cycle at the matrix-calculus level is what makes frameworks like PyTorch's autograd feel like a natural extension rather than a black box: autograd is simply automating the recursive $\boldsymbol{\delta}^{[l]}$ computation derived above.
Additional Help
- 3Blue1Brown's Neural networks Playlist – Watch this playlist. Very good for visualization