I am comparing a half precision workload (tensors are torch.HalfTensor) to a full precision workload (tensors are torch.FloatTensor) by using a command line profiling tool from NVIDIA which reports FLOPS.
I noticed that even when running the model with half precision tensors, the tool reports that only single precision operations happened, albeit many less FLOPS than occured with the full precision model.
My suspicion is that there are some bitpacking optimizations made occuring somewhere in the pytorch/cuda library bindings.
I’d like to confirm this, or if thats not right, hear what the explanation is. Thanks!