If you use torch.no_grad you are disabling the autograd recording of things for backpropagation, so you would have a hard time reducing memory use further (one idea could do (part of?) the teacher computation in fp16 instead of fp32), but you have to watch out for overflows.
In the student, you could use checkpointing (i.e. keeping less activations between forward and backward and recomputing them) to save memory.
Best regards
Thomas