Hmm if that’s the case, it’d mean that majority of the GPU allocation is for the intermediate stages of forward / backward passes? Hence, will it be likely that I can improve the GPU allocation if I reduce the data types of the inputs? e.g. converting from float 64 to float 32.