Does this depends on the method that created the OOM error?
Also could it be that when the python error is raised, it gets a ref to every local objects where the error occurred. I am not sure how these objects behave but they might be holding onto some tensors.