Linear Supervised Learning Series¶

Part 3: The perceptron¶

In this post we discuss the perceptron - a historically significant and useful way of thinking about linear classification.

Press the button 'Toggle code' below to toggle code on and off for entire this presentation.

1.1 The perceptron cost function¶

With two-class classification we have a training set of $P$ points $\left\{ \left(\mathbf{x}_{p},y_{p}\right)\right\} _{p=1}^{P}$ where $y_p$ 's take on just two label values from $\{-1, +1\}$ .

As we saw with logistic regression the decision boundary was formally given as a hyperplane

$\begin{equation} w_0+\mathbf{x}^{T}\mathbf{w}^{\,} = 0 \end{equation}$

Lets look at the expression $\text{max}\left(0,\,-y_{p}\left(w_0+\mathbf{x}_{p}^{T}\mathbf{w}\right)\right)$ more closely:

It is always non-negative: it returns zero if $\mathbf{x}_{p}$ is classified correctly, and returns a positive value if classified incorrectly.

It is convex: because it is the composition of ReLU (itself a convex function) and a linear function.

We sum up this expression over the entire dataset, giving the so-called perceptron or ReLU cost

$\begin{equation} g\left(w_0,\mathbf{w}\right)=\underset{p=1}{\overset{P}{\sum}}\text{max}\left(0,\,-y_{p}\left(w_0+\mathbf{x}_{p}^{T}\mathbf{w}\right)\right) \end{equation}$

Example 1: Softmax approximation to the ReLU¶

The ReLU function

$\begin{equation} g\left(s\right)=\mbox{max}\left(0,\,s\right) \end{equation}$

and its smooth softmax approximation

$\begin{equation} g\left(s\right)=\mbox{soft}\left(0,\,s\right)=\mbox{log}\left(1+e^{s}\right) \end{equation}$

Linear Supervised Learning Series¶

Part 3: The perceptron¶

1. The perceptron¶

1.1 The perceptron cost function¶

Example 1: Using gradient descent to minimize the ReLU cost¶

1.2 The smooth softmax approximation to the ReLU perceptron cost¶

Example 1: Softmax approximation to the ReLU¶

1.3 Predictions, misclassifications, and accuracy with the perceptron cost¶