As far as I understand the issue, your code runs fine using batch_size=5 and only a single step, but runs out of memory for multiple steps using batch_size=1.
Is this correct?
If so, are you sure the forward and backward passes are actually called?
Could you remove the condition checking the label batch dimension?