Question: While using "Two class neural network" in Azure ML, I encountered "Momentum" property. As per [documentation](https://docs.microsoft.com/en-us/azure/machine-learning/studio-module-reference/neural-network-regression), which is not clear, it says

> 
> For The momentum, type a value to apply during learning as a weight on
> nodes from previous iterations.
> 
> 
> 

Although that is not very clear. Can someone please explain?

Answer: 
Momentum in neural networks is a **variant of the *stochastic gradient descent***. It replaces the gradient with a *momentum* which is an aggregate of gradients as very well explained [here](https://towardsdatascience.com/10-gradient-descent-optimisation-algorithms-86989510b5e9).

It is also the common name given to the *momentum factor*, as in your case.

**Maths**

The momentum factor is a coefficient that is applied to an extra term in the weights update:
[![Momentum](https://i.stack.imgur.com/WTOTg.jpg)](https://i.stack.imgur.com/WTOTg.jpg)

*Note*: image from [visual studio magazine post](https://visualstudiomagazine.com/articles/2017/08/01/neural-network-momentum.aspx#:%7E:text=Neural%20network%20momentum%20is%20a,known%2C%20correct%2C%20target%20values.)

**Advantages**

Beside others, momentum is known to speed up learning and to help not getting stuck in local minima.

**Intuition behind**

As it is really nicely explained in [this quora post](https://www.quora.com/What-is-an-intuitive-explanation-of-momentum-in-training-neural-networks), the momentum comes from physics:

> 
> Momentum is a physical property that enables a particular object with
> mass to continue in it's trajectory even when an external opposing
> force is applied, this means overshoot. For example, one speeds up a
> car and then suddenly hits the brakes, the car will skid and stop
> after a short distance overshooting the mark on the ground.
> 
> 
> The same
> concept applies to neural networks, during training the update
> direction tends to resist change when momentum is added to the update
> scheme. When the neural net approaches a shallow local minimum it's
> like applying brakes but not sufficient to instantly affect the update
> direction and magnitude. Hence the neural nets trained this way will
> overshoot past smaller local minima points and only stop in a deeper
> global minimum.
> 
> 
> Thus momentum in neural nets helps them get out of
> local minima points so that a more important global minimum is found.
> Too much of momentum may create issues as well as systems that are not
> stable may create oscillations that grow in magnitude, in such cases
> one needs to add decay terms and so on. It's just physics applied to
> neural net training or numerical optimizations.
> 
> 
> 

**In video**

[This video](https://www.youtube.com/watch?v=qhWperPtnas) shows a backpropagation for different momentum values.

**Other interesting posts**

[How does the momentum term for backpropagation algorithm work?](https://cs.stackexchange.com/questions/31874/how-does-the-momentum-term-for-backpropagation-algorithm-work)

Hope it helps.
Momentum is a technique to prevent sensitive movement. When the gradient gets computed every iteration, it can have totally different direction and the steps make a zigzag path, which makes training very slow. Something like this.

[![enter image description here](https://i.stack.imgur.com/epW89.jpg)](https://i.stack.imgur.com/epW89.jpg)

To prevent this from happening, momentum kind of stabilizes this movement.
You can find more in the [Following Article](https://ai-pool.com/a/s/optimization-methods--gradient-descent)