Is the memory used on all GPUs or just the two you’ve selected?
Sometimes it can be problematic to set the env variable inside the python script, e.g. when CUDA was already initialized. The workaround would be to set the available GPUs before calling the script:
CUDA_VISIBLE_DEVICES=2,3 python script.py

How did you check your data is not fed into the model?