If your Tensors are on GPU, all operation will happen on the GPU.
Note that it is cuda is known to have a high initial memory usage (it’s tracked here 5).
It is simple to get minimal memory usage during inference by wrapping your inference code into a with torch.no_grad(): context.