Many thanks!
I’ve found this solution, and thought that it doesn’t work, but more accurate check shows, that it works, but with some additional porps.
It doesn’t see the card, which is in order after the dropped one (3 is invisible, 2 is dropped, 0 and 1 are ok).
test_gpu.py:
import torch
print(torch.cuda.is_available())
print(torch.cuda.device_count())
print(torch.zeros(10, 10).to(torch.device('cuda:2')))

$>export CUDA_VISIBLE_DEVICES=0,1,3
$>python test_gpu.py
True
2
Traceback (most recent call last):
  File "test_gpu.py", line 4, in <module>
    print(torch.zeros(10, 10).to(torch.device('cuda:2')))
RuntimeError: CUDA error: invalid device ordinal

Do you have any ideas, how to fix it?