I set the network to generate weights randomly following normal distribution of mean = 0 and Stdev =1, trained it at first with Adam. then i saved the weights and went on with training using SGD, and thats when i started to see this wavy loss, so I assume its form the momentum and the learning rate.
(edit) worth mentioning that, I got this wavy loss without changing the learning rate and the momentum, i just their values and run the model for a 1000 epoch and i got the curve above.