Hi,
thanks for the reply!
Indeed I finally found out that the padding works correctly and that it was due to the hyperparameter settings that the network did not learn anything for batch sizes higher than 1.
I originally used the same hyperparameters as in a Theano implementation and therefore assumed that the same hyperparameters would also work for my PyTorch implementation.
For me, I had to add gradient clipping with a threshold of 2.0 (previously I had no gradient clipping) for the network to learn with lr=1e-3 and batch sizes between 1 and 64 (did not test higher).
lr=1e-4 would work without gradient clipping for a batch size of 1 and 10 with and without gradient clipping but it would not work with higher batch sizes (again with and without gradient clipping).
But I think these results are very task-dependent.
The important thing I learned from this is to also consider the hyperparameters if my model fails to learn for higher batch sizes.