Yes that’s true. Actually, the case I have in mind involves tensor-creating functions.
I used to pass a device argument to all such functions, which became cumbersome when there were a lot and they were nested and things.
def make_tensor_device(device): 
    return torch.tensor([1.,2.]).to(device)
   
device = torch.device(f"cuda:{ARGS.device}")
t = make_tensor_device(device)

The context manager simplified this because I can just set the default gpu and then call .cuda() on created tensors.
def make_tensor():
    return torch.tensor([1.,2.]).cuda() 

device = torch.device(f"cuda:{ARGS.device}")  
with torch.cuda.device(device):
    t = make_tensor()

Now it seems I have to go back to using the device argument. Not the end of the world, but it would be nice to allow setting the default device to either the cpu or a specific gpu, maybe that would even mean that .cuda() would no longer be necessary.