The “CUDA out of memory” issue is raised, if your GPU is running our of memory and is unrelated to the storage of numpy arrays.
Try to reduce the batch size for your input tensors to reduce the memory usage or alternatively use torch.utils.checkpoint to trade compute for memory.