The code is a fine tuning job for a chatbot with over 400M parameters (BlenderBot2, see ParlAI/projects/blenderbot2 at main · facebookresearch/ParlAI · GitHub 1) so it is not amenable to creating a code snippet to show the issue.  Is there another way to provide useful information?
Here is the env info…
(ParlAI) C:\Users\Steve>python -m torch.utils.collect_env
Collecting environment information…
PyTorch version: 1.9.0
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A
OS: Microsoft Windows 10 Home
GCC version: Could not collect
Clang version: Could not collect
CMake version: version 3.19.1
Libc version: N/A
Python version: 3.8 (64-bit runtime)
Python platform: Windows-10-10.0.19042-SP0
Is CUDA available: True
CUDA runtime version: 11.2.142
GPU models and configuration: GPU 0: GeForce RTX 2070 SUPER
Nvidia driver version: Could not collect
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.2\bin\cudnn_ops_train64_8.dll
HIP runtime version: N/A
MIOpen runtime version: N/A
Versions of relevant libraries:
[pip3] numpy==1.20.3
[pip3] pytorch-ranger==0.1.1
[pip3] torch==1.9.0
[pip3] torch-optimizer==0.1.0
[pip3] torchtext==0.10.0
[conda] blas                      2.106                       mkl    conda-forge
[conda] cudatoolkit               11.1.1               heb2d755_7    conda-forge
[conda] libblas                   3.9.0                     6_mkl    conda-forge
[conda] libcblas                  3.9.0                     6_mkl    conda-forge
[conda] liblapack                 3.9.0                     6_mkl    conda-forge
[conda] liblapacke                3.9.0                     6_mkl    conda-forge
[conda] mkl                       2020.4             hb70f87d_311    conda-forge
[conda] numpy                     1.21.1           py38h09042cb_0    conda-forge
[conda] pytorch                   1.9.0           py3.8_cuda11.1_cudnn8_0    pytorch
[conda] pytorch-ranger            0.1.1                    pypi_0    pypi
[conda] torch                     1.9.0                    pypi_0    pypi
[conda] torch-optimizer           0.1.0                    pypi_0    pypi
[conda] torchtext                 0.10.0                   pypi_0    pypi