Using ADAM:-
loss 1.1901756525039673
iteration 1
loss 400.5993347167969
iteration 2
loss 78.54071807861328
iteration 3
loss 191.35740661621094
iteration 4
loss 153.02627563476562
iteration 5
loss 29.623409271240234
iteration 6
loss 7.654780864715576
iteration 7
loss 4.365676403045654
iteration 8
loss 24.920808792114258
iteration 9
loss 3.465801954269409
iteration 10
loss 1.6431617736816406
using SGD
iteration 0
loss 1.0905007123947144
iteration 1
loss 1.0449053049087524
iteration 2
loss 1.0095171928405762
iteration 3
loss 0.9810587167739868
iteration 4
loss 0.95912104845047
iteration 5
loss 0.950375497341156
iteration 6
loss 0.9360883235931396
iteration 7
loss 0.9562820196151733
Can you explain why in the beginning, loss is increasing so high in ADAM :