I followed the tutorial: AUTOMATIC MIXED PRECISION 1 and found it can’t reduce the network’s memory footprint.
Here are the hyperparams used and the results:
batch_size = 512
in_size = 4096
out_size = 4096
num_layers = 20
num_batches = 100
epochs = 3

=== results ===
Default precision:
Total execution time = 15.974 sec
Max memory used by tensors = 4773790720 bytes

Mixed precision:
Total execution time = 13.834 sec
Max memory used by tensors = 5268703744 bytes

The calculations are on GPU (RTX 3080).
I can’t understand why mixed precision use more memory than default precision.