First of all, thank you for your response.
I have tried your approach. However, I still need so many variables in my script, so I cannot do that, as the programm wouldn’t work as a consequence. If I only use “torch.cuda.empty_cache()”, I still get “CUDA out of memory”, so aparently I cannot solve it with that.
EDIT:
This crashes:
        outputs = self._model(inputs)
        outputs2 = self._model(inputs)
        outputs3 = self._model(inputs)
        outputs4 = self._model(inputs)
        outputs5 = self._model(inputs)
        outputs6 = self._model(inputs)
        outputs7 = self._model(inputs)
        outputs8 = self._model(inputs)
        outputs9 = self._model(inputs)

While this not:
        outputs = self._model(inputs)
        outputs = self._model(inputs)
        outputs = self._model(inputs)
        outputs = self._model(inputs)
        outputs = self._model(inputs)
        outputs = self._model(inputs)
        outputs = self._model(inputs)
        outputs = self._model(inputs)
        outputs = self._model(inputs)

So it is clearly about the memory used to allocate the output tensors in the GPU. I have tried to first replicate the outputs in a CPU tensor before computing more samples, but it is still not working.