Solved by caped-vigilante in post #5 
              
              
                batch size of 4 or above, is a good start. It all depends upon how much GPU memory you have. Bigger the batch size, the training is more stable. 
For the batch normalization part, you can try this: 
class Netz(nn.Module):
    def __init__(self):
        super(Netz, self).__init__()
        self.conv…