In that case your initial code doesn’t seem to have used architecture-specific kernels.
You could trade compute for memory by using e.g. torch.utils.checkpoint or by lowering the batch size, if this fits your use case.