Could you check the memory usage of the used modules by printing either torch.cuda.memory_allocated() or torch.cuda.memory_summary(). Based on these results you could then check if the memory requirement is expected or not.