Thanks for the suggestion. I tried overfitting the model on a small data set and it wasn’t able to do so. I also tried using adam instead of SGD, but that didnt change anything.
How can i change my architecture to actually make this network learn?