Since this seems like a memory limitation imposed by PyTorch, feel free to file a GitHub issue over at https://github.com/pytorch/pytorch/issues. It would be valuable to have a repro where extra memory is allocated unexpectedly.