legoh:

it’d mean that majority of the GPU allocation is for the intermediate stages of forward / backward passes?


That might be the case.
E.g. for a single conv layer, the output might contain more elements than the input, if out_channels > in_chanels and you don’t reduce the spatial size.



 legoh:

converting from float 64 to float 32.


Yes. PyTorch uses float32 by default, so if you’ve called double() on your model (and don’t strictly need the precision), I would recommend to use FP32.