torch.cuda.empty_cache() will only remove all reserved and unused memory, which the custom caching allocator in PyTorch holds to avoid expensive memory allocations. The CUDA context won’t be freed.
If I understand the map_location argument with a dict, it would remap the tensors from the key location to the value location, so both GPUs would be initially used (which should thus also create a context on both of them).
To avoid it you could store the state_dict on the CPU and map it to the desired GPU afterwards.