My batch size is constant and equal to 10.
My learning rate starts at 1e-3 and I’m using decay:
	optim_scheduler = optim.lr_scheduler.StepLR(optimizer, 10, 0.989)
[...]
	optim_scheduler.step()

The architecture that I’m trying is pretty much Convolutional Layers followed by Max Pool layers (the last one is an Adaptive Max Pool), using ReLU and batch normalization. I have a GRU layer and a fully connected using a single hidden layer.
My inputs are variable sized arrays that were padded inside the batch. I’m padding as less as possible since I sort the dataset by the length of the array.
I’m training only for a small number of epochs since the error is weird, but I believe that it would keep increasing.
(Following something I found in the forum, I added the parameter amsgrad=True in my Adam optimizer, but I still have this loss problem)