OK, thanks to the downgrade, I found out the problem was that I was applying an nn.Linear layer contained in my GPU to an input tensor that was stored in my CPU. My guess is, the operation went through without errors, this corrupted the memory in the GPU and, when I tried to access a random variable in my code that was stored in the GPU, the corrupted memory ended up throing this error. PyTorch 1.5.0 actually gave me a descriptive error and I was able to fix this.
Then, I went back to PyTorch 1.6.0 and the cudaErrorIllegalAddress error was gone.