If when OOM occurs you are still holding onto references (explicitly or internally) to graphs and/or tensors, they will continue occupy memory. Occupied memory is not free-able by empty_cache().
Perhaps manually del the relevant variables will help. But I’m not sure catching OOM like this won’t have other downsides.