The problem is that even if labels are not traced, you are enlisting them and thus, gpu memory keeps increasing until you reach a limit (which is the amount of iterations per epoch). If you reduce batch size you should probably observe even more memory comsumption as more labels would be allocated.
Anyway that does not explains the slow down. Is it possible that your machine gets slowed down by any other reason? (Dataloader for example?) could you split the timing into data loading and gpu computation?  For example, you could get ram full and start using swap memory or something like that.
At least in the snippet you pasted I cannot apreciate more errors than the already mentioned ones. If you properly detached tensors after my suggestion, the erroy may come from other side.
In an extreme case, you should try to generate data randomly and un a minimal example with just the model and simple training pipe to discard issues in the model.
If it does work (which will probably will), start to save those variables (accuracy and so on). If it does work, check dataloader.