Follow-up: On the RTX30 Series you MUST use at least CUDA 11.1 since it’s the first CUDA version which offers support for NVIDIA Ampere architecture (source: CUDA 11.1 Release Notes chapter 1.2).
After installing CUDA Toolkit 11.3 and LibTorch for CUDA 11.3 the model started working properly.