Hi,
I’m using a bash script to do grid-search for my pytorch code.
Something like as follows:
for lr in 0.1 0.01 0.001
do
    for wd in 0.1 0.01 0.001
    do
       # call pytorch script for the current setting of hyperparams
       python my_pytorch_script.py --lr=$lr --wd=$wd
    done
done

I noticed that, sometimes, a process finishes without freeing the GPU memory. It eventually causes the GPU to go OOM even though I execute each program sequentially.
How can I fix this?