Computational Calculus Series

Part 7: The anatomy of a hyperplane

In this post we describe important characteristics of the hyperplane including the concept of the direction of steepest ascent. These concepts are fundamental to the notion of mult-input derivatives (the gradient), gradient descent, as well as linear regression and classification schemes.

Press the button 'Toggle code' below to toggle code on and off for entire this presentation.

from IPython.display import display
from IPython.display import HTML
import IPython.core.display as di # Example: di.display_html('<h3>%s:</h3>' % str, raw=True)

# This line will hide code by default when the notebook is exported as HTML
di.display_html('<script>jQuery(function() {if (jQuery("body.notebook_app").length == 0) { jQuery(".input_area").toggle(); jQuery(".prompt").toggle();}});</script>', raw=True)

# This line will add a button to toggle visibility of code blocks, for use with the HTML export version
di.display_html('''<button onclick="jQuery('.input_area').toggle(); jQuery('.prompt').toggle();">Toggle code</button>''', raw=True)

Toggle code

1. Building hyperplanes from simple pieces

In this Section we describe how to construct complex hyperplanes from relatively simple parts. Along the way we will also see several important concepts - most notably the idea of the direction of steepest ascent arisess naturally from the notion of slope.

1.1 Single input hyperplanes

The formula for a line

\begin{equation} g(w) = a + bw \end{equation}

• $a$ = point at which it strikes or intersects the vertical axis

• $b$ = and the steepness or slope of that line

• below we plot $g(w) = 2w$ and $g(w) = -w + 2$

# create two quadratic functions
func1 = lambda w: 2*w
func2 = lambda w: -w + 2

# use custom plotter to show both functions
title1 = '$g(w)=$2 + 3w$'; title2 = '$g(w)=$-w$';
callib.plotter.double_2d_plot(func1 = func1, func2 = func2,title1 = title1,title2=title2,fontsize = 13,color = 'lime')

• we can visualize the slope of a single line using a horizontal vector with magnitude $b$

• as a direction the slope is often referred to as the direction of steepest ascent

• shows a) the direction in which the line is increasing and b) how quickly it is increasing

• same logic: steepest descent direction of line is negative slope $-b$

# animate 2d slope visualizer
func = lambda w: 2 + 3*w
callib.slope_visualizer.animate_visualize2d(func = func,num_frames = 50)

• in three dimensions have two input variables $w_1$ and $w_2$, but can form a similar equation using a single input

• for example, the formula

\begin{equation} g(w_1,w_2) = a + bw_1 \end{equation}

• takes in the first input $w_1$ and treats it like a line, outputing $a + bw_1$.

• this hyperplane - like the corresponding line - has a stmeepness or slope given by $b$ (no defined over two dimensions)

• below we plot two and three dimensional versions of the line $g(w_1,w_2) = 2-w_1$ side by side

# plot a single input quadratic in both two and three dimensions
func1 = lambda w: 2-2*w 
func2 = lambda w: 2-2*w[0] 

# use custom plotter to show both functions
title1 = '$g(w)=$2-2w$'; title2 = '$g(w_1,w_2)=2-2w_1$';
callib.plotter.double_2d3d_plot(func1 = func1, func2 = func2,title1 = title1,title2=title2,fontsize = 18,color = 'lime')

• like one dimensional example, here we can visualie the directions of steepest ascent and descent given by the individual direction slope of the hyperplane - as a vector in the input space

• for example, with the previous example of $g(w_1,w_2) = 2-2w_1$ this ascent direction is $\left(b_1,0\right) = (-2,0)$

• descent direction is then $-\left(b_1,0\right) = (2,0)$.

• below animate a hyperplane and these vectors - the ascent vector colored blue, the descent color in red - along range of values for $b_1$

# define hyperplane
func = lambda w:  2-2*w[0]

# animate 2d slope visualizer
callib.slope_visualizer.animate_visualize3d(func=func,num_frames=50)

• of course we can define this single input hyperplane along any dimension we want.

• in general if we have $N$ possible inputs $\mathbf{w}=[w_1,\,\,w_2,\,\,\cdots\,w_N]$ we can define it along the $n^{th}$ dimension as $g(\mathbf{w}) = a + bw_n$.

1.2 Constructing general hyperplanes when $N=2$

• we can form more complex hyperplanes by summing up a number of single input ones like those discussed above

• for example, with $N=2$ inputs if we form the two single input hyperplanes

\begin{array} \ g_1(w_1,w_2) = a_1 + b_1 w_1 \\ g_2(w_1,w_2) = a_2 + b_2 w_2 \\ \end{array}

one along each input dimension.

\begin{array} \ g_1(w_1,w_2) = a_1 + b_1 w_1 \\ g_2(w_1,w_2) = a_2 + b_2 w_2 \\ \end{array}

• summing these gives us a more complex hyperplane $g(w_1,w_2) = g_1(w_1,w_2) + g_2(w_1,w_2) = \left( a_1 + a_2 \right) + b_1w_1 + b_2w_2$

• this has a slope along each input dimension explicitly controlled by its corresponding single input hyperplane

For example, below we plot the following single input hyperplanes

\begin{array} \ g_1(w_1,w_2) = 1-2w_1 \\ g_2(w_1,w_2) = 1 + 2w_2 \\ \end{array}

as well as their sum $g(w_1,w_2) = 2 - 2w_1 + 2w_2$

\begin{equation} g(w_1,w_2) = 2 - 2w_1 + 2w_2 \end{equation}

• what is the steepest ascent direction of the complex hyperplane?

• to get this we sum the steepest ascent directions of both hyperplanes, giving $\left(b_1,b_2\right) = (-2,2)$

• likewise hte descent direction is just the negative of this: $-\left(b_1,b_2\right) = (2,-2)$

# plot a single input quadratic in both two and three dimensions
func1 = lambda w: 1 + 2*w[0] 
func2 = lambda w: 1 - 2*w[1] 
func3 = lambda w: 2 + 2*w[0] -2*w[1]  

# use custom plotter to show both functions
view = [25,-50]
callib.slope_visualizer.visualize3d(func1 = func1,func2=func2,func3=func3,view = view)

• below we animate a range of complex hyperplanes starting with the one above

# define hyperplane
func = lambda w:  2 -2*w[0] - 2*w[1] 

# animate 2d slope visualizer
callib.slope_visualizer.animate_visualize3d(func=func,num_frames=50)

• this simple method for constructing multi-input hyperplanes can be used to produce every possible multi-input hyperplane in two dimensions

• the analagouos procedure can also generate every possible hyperplane in higher dimensions as well.

• more generally for $N$ dimensional input we can define a single input hyperplane along each dimension

\begin{array} \ g_1(\mathbf{w}) = a_1 + b_1 w_1 \\ g_2(\mathbf{w}) = a_2 + b_2w_2 \\ \,\,\,\,\,\,\,\,\,\,\,\,\,\,\, \vdots \\ g_N(\mathbf{w}) = a_N + b_N w_N \\ \end{array}

• summing them up as $g(\mathbf{w}) = \sum_{n=1}^N g_n(\mathbf{w})$ gives

\begin{equation} g(\mathbf{w}) = (a_1 + a_2 + \cdots a_N) + (b_1 w_1 + b_2 w_2 + \cdots b_N w_N) \end{equation}

\begin{equation} g(\mathbf{w}) = (a_1 + a_2 + \cdots a_N) + (b_1 w_1 + b_2 w_2 + \cdots b_N w_N) \end{equation}

• likewise denoting $a = \sum_{n=1}^{N} a_n$ and and the $\mathbf{b}$ the $N\times 1$ vector

\begin{equation} \mathbf{b} = \begin{bmatrix} b_1 \\ b_2 \\ \vdots \\ b_N \end{bmatrix} \end{equation}

• write hyperplane compactly

\begin{equation} g(\mathbf{w}) = a + \mathbf{w}^T\mathbf{b} \end{equation}

\begin{equation} g(\mathbf{w}) = a + \mathbf{w}^T\mathbf{b} \end{equation}

• here direction of steepest ascent given by $\mathbf{b}$, steepest descent direction is $-\mathbf{b}$