Type conversions are “differentiable” as can be seen in this dummy mixed-precision example:
x = torch.randn(1, 10, dtype=torch.float16, device='cuda')
w1 = torch.randn(10, 1, requires_grad=True, dtype=torch.float16, device='cuda')
w2 = torch.randn(1, 1, requires_grad=True, dtype=torch.float32, device='cuda')

output = torch.matmul(x, w1)
output = output.float()
output = torch.matmul(output, w2)

loss = (output - torch.randn(1, 1, dtype=torch.float32, device='cuda'))**2
loss.backward()

print(w1.grad)
print(w2.grad)

As you can see, I’m starting with float16 input and parameters, and convert them to float32 later, which just works fine, as Autograd just transforms the gradients back to the appropriate type.