I met this problem as well. This problem is solved by setting pin_memory=False.
Interestingly, my PC should be classified as sufficient VRAM/RAM to play with. Dont know why so laggy when pin_memory = True.
The environment is listed as follows:

Environment
CPU:
i9-12900K
RAM:
64GB DDR5 5200MHz

CUDA:


GPU:
NVIDIA GeForce RTX 3090
NVIDIA GeForce RTX 3090
available: True
version: 11.2


Packages:


numpy: 1.21.5
pyTorch_debug: False
pyTorch_version: 1.10.0
pytorch-lightning: 1.5.8
tqdm: 4.62.3


System:


OS: Linux
architecture:
64bit
ELF
processor: x86_64
python: 3.9.7
version: #202201071026-Ubuntu SMP Fri Jan 7 16:52:09 UTC 2022