Neural Networks From Scratch With Buzz Lightyear (Part 3: Multi-Layer Perceptrons)

In the previous article, we learned how to make a single perceptron find a best-fit line, with an introduction to complex models and activation functions. Here we’ll expand into a fully-connected multi-layer neural network by chaining individual perceptrons. We accomplish this by combining gradient descent with backpropagation.

A multi-layer perceptron (MLP) is composed of layers, each containing some perceptrons. When we input data, each layer calculates an output based on its perceptrons’ weights, but instead of passing those outputs as the final answer, it passes them to the next layer until the last. This is a forward pass. The benefit of all the convolution is many connections, which means a lot of complexity. As we’ve seen, complexity is useful in a neural net — it lets us fit any kind of data.

Below is a visual MLP. Buttons choose the number of layers and perceptrons per layer. Press any perceptron to see how the forward pass flows from it.

The number of perceptrons can change from layer to layer. Some very useful variants — autoencoders and GANs — depend on this property.

Because there are so many weights, biases and inputs in a neural network, manipulating them as individual values is impractical. ML libraries (and us, starting now) use arrays — more accurately matrices and tensors — to represent an MLP. For simplicity every matrix here will be one-dimensional, and all matrix multiplication is element-wise:

(x_{1} x_{2}) \cdot (y_{1} y_{2}) = (x_{1} \cdot y_{1} x_{2} \cdot y_{2})

Each perceptron will be a matrix of weights (one per input) plus a bias. We find the output of a perceptron by multiplying it with the input matrix and adding the bias:

\sum ((w_{1} w_{2}) \cdot (x_{1} x_{2})) + b = w_{1} \cdot x_{1} + w_{2} \cdot x_{2} + b

A layer is an array of perceptrons; an MLP is an array of layers. We can now manipulate thousands of parameters in one call. Hover over each perceptron below to see how many parameters it has.

// a perceptron is an array of weights
class Perceptron extends Array {
    constructor(numInputs) {
        for (var i = 0; i < numInputs; i++) this.push(random(-1, 1))
        this.bias = random(-1, 1)
    }
    eval(d) { return sum(this * d) + this.bias }
}

// a layer is an array of perceptrons
class Layer extends Array {
    constructor(numPerceptrons, numInputs) {
        super()
        for (var i = 0; i < numPerceptrons; i++) {
            this.push(new Perceptron(numInputs))
        }
    }
    eval(d) {
        var res = []
        this.lastInput = d
        for (var i = 0; i < this.length; i++) res.push(this[i].eval(d))
        this.lastOutput = res
        return res
    }
}

// an MLP is an array of layers
class MLP extends Array {
    constructor(schema) {
        super()
        for (var i = 1; i < schema.length; i++) {
            this.push(new Layer(schema[i], schema[i - 1]))
        }
    }
}

// first element describes the incoming data
// the rest describe the number of perceptrons per layer
var schema = [1, 1, 2, 3, 4]
var mlp = new MLP(schema)

With our MLP components in code, we can implement the forward pass and see how it maps out on a graph.

MLP.forward = function (input) {
    // feed the output of each layer as input of the next
    for (var i = 0; i < this.length; i++) input = this[i].eval(input)
    return input
}

var schema = [1, 1, 2, 3, 4]
var mlp = new MLP(schema)

You might be thinking — “isn’t the whole point of an MLP that it’s more complex than a line? What’s the difference between this and a single perceptron?”. Correct. Nesting perceptrons is just applying transformations on a line; it doesn’t change the shape of the graph. This is where ReLU comes in. ReLU is an activation function (like sigmoid) that takes a line and clips off its negative side.

It might seem trivial, but it’s enough to let our MLP take shapes far more complex than quadratics and cubics — each perceptron can now change the direction of the line. The derivative of ReLU is also convenient: 1 for the line segment, 0 for the flattened segment.

// don't make the last layer a ReLU — we want negative outputs allowed
var schema = [1, [1,'relu'], [2,'relu'], [3,'relu'], 4]
var mlp = new MLP(schema)

A quick calculus refresher. Derivatives are slopes of curves at a point. Like we say $m = \frac{Δ y}{Δ x}$ for a line, we can say $m = \frac{\partial y}{\partial x}$ for any graph. Our cost function is $c (x) = (x - y)^{2}$ , its derivative in terms of $x$ is $2 \cdot (x - y)$ , and our perceptron is $p (x) = w \cdot in p u t + b$ . The cost would be $c (p (x))$ and we want $\frac{\partial c ( p ( x ))}{\partial w}$ . We only know it in terms of $p (x)$ , but using the property $\frac{y}{x} \cdot \frac{x}{z} = \frac{y}{z}$ we can rephrase as $\frac{\partial c ( p ( x ))}{\partial p ( x )} \cdot \frac{\partial p ( x )}{\partial w}$ which simplifies to $2 \cdot (p (x) - y) \cdot in p u t$ . The chain rule.

Backpropagation uses the chain rule to calculate derivatives for perceptrons within perceptrons — perceptrons that feed into other perceptrons. Two perceptrons joined end to end:

p_{1} = w_{1} \cdot x + b_{1} p_{2} = w_{2} \cdot p_{1} + b_{2} cos t = (p_{2} - y)^{2}

Easy one first — $p_{2}$ : $\frac{\partial cos t}{\partial p _{2}} = 2 \cdot (p_{2} - y)$ . For $p_{1}$ via chain rule:

\frac{\partial cos t}{\partial p _{1}} = \frac{\partial cos t}{\partial p _{2}} \cdot \frac{\partial p _{2}}{\partial p _{1}} = 2 \cdot (p_{2} - y) \cdot w_{2}

The derivative of one perceptron depends on the one immediately next to it. Extending to three:

\frac{\partial cos t}{\partial p _{3}} = 2 \cdot (p_{3} - y)

\frac{\partial cos t}{\partial p _{2}} = 2 \cdot (p_{3} - y) \cdot w_{3}

\frac{\partial cos t}{\partial p _{1}} = 2 \cdot (p_{3} - y) \cdot w_{3} \cdot w_{2}

What happens when a perceptron feeds into two instead of one?

Since $p_{1}$ shows up in both $p_{2}$ and $p_{3}$ , we sum up the derivatives:

\frac{\partial cos t}{\partial p _{1}} = \frac{\partial cos t}{\partial p _{2}} \cdot \frac{\partial p _{2}}{\partial p _{1}} + \frac{\partial cos t}{\partial p _{3}} \cdot \frac{\partial p _{3}}{\partial p _{1}}

When this is scaled up, any perceptron’s derivative can be calculated from the derivatives of the ones it feeds into. It’s the forward pass in reverse.

MLP.backward = function (desired) {
    // calculate cost derivatives for the last layer, multiplied with activation derivs
    var global_derivs = cost_deriv(this[this.length-1].lastOutput, desired)
                        * this[this.length-1].activation_derivs()
    var prev_derivs;
    for (var i = this.length - 1; i >= 0; i--) {
        var l = this[i]
        var d = this.derivs[i]    // clone of layer for derivatives
        if (i != 0) prev_derivs = new Matrix(this[i-1].length, 0)

        for (var j = 0; j < d.length; j++) {
            var p = new Matrix(d[j].length, global_derivs[j]) * l.lastInput
            d[j] = d[j] + p
            d[j].bias += global_derivs[j]
            if (i == 0) continue
            for (var k = 0; k < prev_derivs.length; k++) {
                prev_derivs[k] += global_derivs[j] * l[j][k]
            }
        }
        global_derivs = prev_derivs
    }
}

MLP.applyDerivs = function () {
    for (var i = 0; i < this.derivs.length; i++) {
        for (var j = 0; j < this.derivs[i].length; j++) {
            for (var k = 0; k < this.derivs[i][j].length; k++) {
                this[i][j][k] -= WEIGHT_LRATE * this.derivs[i][j][k]
            }
            this[i][j].bias -= BIAS_LRATE * this.derivs[i][j].bias
        }
    }
    this.clearDerivs()
}

We have a fully operational MLP. We can change it to logistic regression by swapping the last layer’s activation function and the cost:

Note: stop it after running, it tends to eat a lot of computing power.

cost_deriv = neglog_cost_deriv
var schema = [2, 4, 6, [8,'relu'], 8, [1,'sigmoid']]
var mlp = new MLP(schema)

A few in-depth problems and solutions, briefly. This article isn’t comprehensive — for deeper coverage, see Andrew Ng, Andrej Karpathy and Geoffrey Hinton on YouTube, plus sentdex’s practical series.

Learning Rate

When implementing gradient descent, I glossed over the important numbers in the code — the learning rates.

xs[i] -= 0.01 * derivative
// ...
perceptron[0] += 0.3 * adj[0]

Don’t just subtract the gradient itself from the weight — it can lead to jumps that overshoot the optimum. Below: how learning rates 1, 0.9, 0.3 and 0.05 work on $x^{2}$ . Press Step (or click the graph) for one pass.

A badly picked rate like 1 hovers above the optimum and never converges. 0.9 keeps overshooting and takes forever. 0.05 wastes time undershooting. Only experience and good guesses get you a good rate — 0.3 in this case. More sophisticated algorithms (RMSProp, AdaMax, Adam) handle this and are used almost ubiquitously in practice, but they’re outside the scope of this article.

Exploding & Vanishing Gradients

In backpropagation, a perceptron’s derivative is the first-layer derivative multiplied by weights from the others. In deep networks this presents a problem. If the weights average ~0.5, a perceptron $x$ layers deep gets multiplied by $0. 5^{x}$ → rapidly zero. Zero gradient → zero gradient descent → the network freezes. This is a vanishing gradient.

If the weights average ~2 we get $2^{x}$ , which blows up. A ten-layer network would have a gradient of 1024. Exploding gradients make a network unstable and can produce “infinite” weights that crash everything.

The fix: initialize weights so gradients get multiplied by reasonable numbers. Normal distributions and Xavier initialization handle this — good overview here.

Regularization

Another problem: overfitting. The model becomes a little too good at predicting the training data at the expense of generalizing. Someone saying Apple stock is going up tomorrow based on the general trend is more credible than someone predicting every minute of tomorrow by memorizing every minute so far.

A common technique: keep the weights as small as possible. Adding the magnitude of the weights to the cost handles this. Good article here.

Batches

Using batches in a neural network means summing the derivatives of many data points before updating the network’s weights. Performance advantage — one backward pass for many forward passes. It also makes intuitive sense — changing the whole network based on one data point is like the government changing because one person said they didn’t like it. Batches are more democratic.

Conclusion

I hope this has helped you understand and implement your own neural networks from scratch. It’s far from comprehensive — consult other sources. The search for knowledge is never over.

To infinity and beyond.