I’ll try to write a very simple explanation to get you an intuition about neural networks and gradients in general.
There are a lot of books, courses etc. that most likely introduce much better to the topic.
First, let’s start with a very simple example.
Imagine we would like to “find” the best weights for the function f(x) = w * x, where we already have some data (x) and targets (f(x) = y). Let’s just assume the ground truth value of our parameter w is 3. In a real use case we wouldn’t know the best values for the parameter, but let’s cheat a bit for this example.
Using the function, we can create a model and just initialize its w to a random number.
We end up with w = 1 as our initial value. This step is referred to as weight initialization.
Now we can calculate the model output using our initial w and compare it to the target we already have. To get a meaningful value of the discrepancy we use a loss function to measure our error. You can imaging the loss function creates a “loss surface”, where for each parameter value, we have a specific loss value.
Let’s just assume our loss function looks like this:
          
            
           1

Using our initial value for w we end up at position 1.
Since our w is a bit too low, we would like to increase it a bit.
So let’s use the derivative (gradient) at the current position to update w.
w = w - learning_rate * dL/dw
Because dL/dw is negative, the new value for w will be bigger than before. In the end we move towards the optimum and end up after the weight update at position 2 in the image.
Because sometimes the derivative is quite large, we could also jump that far, that we miss the optimum and end up on the other side of the loss function (right of the optimum).
This is why we scale the derivative with a learning rate to scale the weight updates. The closer you are to your optimum, the lower your learning rate should be. This methodology is implemented in e.g. ReduceLROnPlateau or other learning rate schedulers.
This is the basic approach for neural networks. The big difference is, that we have a lot of parameters (sometimes millions), which are connected by non-linearities. To calculate the gradient of the loss with respect to all parameters, we can use the back propagation.
The loss surface is also dependent on the parameters and thus high-dimensional.
In the simple example we just used one weight, which created this nice looking loss function.
In reality we would have something like a 1,000,000-dimensional loss surface, which is quite hard to imaging.
I hope it makes things a bit clearer. I’ve heard a some good things about the fast.ai 7 course, so maybe you should check it out. I’m sure they introduce the topic much better.