A batch size of 2 might be too small for batch norm layers, so you should either increase the batch size or maybe even remove the batch norm layers.
I cannot find anything obviously wrong in the code, so I would recommend to play around with the hyper-parameters more (such as removing the scheduler, changing the model architecture etc.).