Gradient Descent


## Steepest descent algorithm
![](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2F75c03b60-3611-43da-85c7-8d73213a3a04%2FUntitled.png?table=block&id=7e8a4390-309f-4b29-99a0-163a7c3073f8&cache=v2)
[Statistical Thinking](https://texonom.com/statistical-thinking-d65f51261f834979a9d6faeb350cff66) U**pdate your probability not change (****[Learning Rate](https://texonom.com/learning-rate-c9a29280131f4f95a6191bde244f6789)****)**
gradient ascent maximize and descent minimize function
Iteratively subtract the derivative of [Loss Function](https://texonom.com/loss-function-e8f6343914494828988137987cf459f9) wrt weight from the weight with the learning rate $\alpha$ and then [Cost Function](https://texonom.com/cost-function-66c21423cc7347909016b3423f2ada7e)  decreases
Repeat until the weight convergence (**[Local extremum point](https://texonom.com/local-extremum-point-f7a7579a29474436b2a8670a28fed921)****)**
### Gradient Descents
|Title|
|:-:|
|[Batch gradient Descent](https://texonom.com/batch-gradient-descent-b25f0157eea74176bec7398fccb73dd5)|
|[Stochastic Gradient Descent](https://texonom.com/stochastic-gradient-descent-d8b8d008e0a34f4bb55175ffba21db44)|
|[Gradient Descent with Momentum](https://texonom.com/gradient-descent-with-momentum-ef703bbc1418412d92ce2487ce880751)|


### Gradient Descent Notion
|Title|
|:-:|
|[Saddle point problem](https://texonom.com/saddle-point-problem-8ba441ca3bad46d6b0b6785305352a50)|
|[Lipschitz Assumption](https://texonom.com/lipschitz-assumption-5c5da7f2f8924237ab56532fc156e2c0)|
|[Momentum Method](https://texonom.com/momentum-method-858e030a03f24243bbfcb59dc6c239dc)|
|[Gradient Clipping](https://texonom.com/gradient-clipping-9d4182db0f0046e3bddc5664e14101b7)|




> [0021 Gradient Descent & Momentum - Deepest Documentation](https://deepestdocs.readthedocs.io/en/latest/002_deep_learning_part_1/0021/)

### Dual number
> [adventures/dual-numbers-and-gradient-descent.ipynb at master · samuelbelko/adventures](https://github.com/samuelbelko/adventures/blob/master/dual-numbers-and-gradient/dual-numbers-and-gradient-descent.ipynb)

> [Online gradient descent written in SQL • Max Halford](https://maxhalford.github.io/blog/ogd-in-sql/)





