Definitions

Input: value that goes in

Weight: a thing that multiplies its input by some value

Perceptron: adds up all the inputs and activates (activation function) after a certain threshold is met

Bias: a constant value added to a node

Simple neural network

Inputs go in → weights are multiplied → only fires after a certain threshold → it then gets added together.

Here is an example implementing a regression model.

Example implements an OR

Example implements an AND

Downsides to a single layer perceptron

The reason we can do AND and OR is because it is binary and linear.

Multi-layer perceptron (MLP)

Note each line has a weight

Multi-layer example

Common Configurations

Stuff on hidden layers

The hidden layers does not normally classify stuff.

They can also represent real world values, or have their own activation functions ect…

Training

Is when you adjust the weights and biases to get the desired output.

Basic idea to adjust the weights (Back propagation)

1. Start with random weights

2. Check how different it is to the desired output

3. Adjust each weight to get better results

4. Check again and repeat

How to adjust the structures topology

1. He didn’t say…. but its clearly imporatant

Gradient Decent

• Gradually move across the plain until youi get lower

• Baso hill climb

Approaches:

• Batch gradient decent (BGD): adjusts all perceptrons at the same time, and for each change will adjust based on all data points, and move to the next lowest point
  • Slow
  • Convergence is guaranteed

• Stochastic gradient decent (SGD): looks at one data point at a time then updates the perceptrons and moves to the next data point
  • Fast
  • Convergence is not guaranteed

Back propagation is a variation of stochastic gradient decent, for multi-layer networks.

Parameter selection

Note the first arg is how many neurons in one layer.
Typically it gets better with more neurons then it slowly gets worse.

How big are the jumps between each epoch (learning cycle).

Activation function. (relu is used in image classification)

Solver.

What can go wrong

• Overfitting, Underfitting

• Gradient decent stuck in local optima

• Vanishing Gradients in deep networks, as the back propagation training works, the later (near the output) layers are changed alot, where as the layers closer to the input are hardly changed