Since some modules are not used during the forward pass, no intermediate tensors would be created. However, the parameters of these unused modules will still be pushed to the device.
If you don’t want to use these modules, you could either remove them from the __init__ or delete them afterwards:
model.multilinear_head = None

The cleaner solution would be the first approach, as the second one might have side effects (e.g. I don’t know, if forward hooks would break etc.).
Alternatively, you could also manually move the unused modules back to the CPU (which would at least save some GPU memory):
model.multilinear_head.cpu()