We had another thread about this at:

  
    
    
    Releasing memory after GPU usage General Discussion
  
  
    Can you check clear_session() doesn't clear memory from GPU · Issue #9379 · keras-team/keras · GitHub ? 
In this thread you can try to close CUDA in python with Numba