I haven’t profiled the memory consumption of this model, but do you have an estimation what memory usage would be expected or why do you think it’s too much?
Also note that nvidia-smi shows the memory used by the CUDA context, the allocated memory, and the cached memory (which can be reused by PyTorch without new allocations).