If you install the binaries with a cudatoolkit, your local CUDA installation won’t be used.
In case your memory increases, make sure to not store any tensors, which are still attached to the computation graph.
This could usually happen, if you are storing the outputs, predictions, or the loss in a list.
To release the device memory, you would need to delete the object (del x) and can clear the cache afterwards.