Hello,
I made a custom module that need high precision (float64) for few operations in the forward pass. Those operations involve few float64 tensors registered as buffer in the module as shown in the following snipped:
class custom_module(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.register_buffer('foo_tensor', get_foo_tensor())
        self.register_buffer('float64_tensor',  get_float64_tensor())

    def forward(self, x):
        input_dtype = x.dtype
        x = some_regular_op(x, self.foo_tensor)

        # Avoid any autocast context messing around here
        with torch.cuda.amp.autocast(enabled=False):
            x = x.to(dtype = torch.float64)
            x = some_high_precision_op(x, self.float64_tensor)

        x = x.to(input_dtype)
        return x

As I registered my tensors, I can easily change the device of my module with .to(device). But now, I would like also to be able to use .to(dtype), but without affecting the precision of float64_tensor, is that possible in some way ?
Furthermore, my custom_module may be used as sub-module inside a model, so I want to be able to perform model.to(dtype) and still preserve the precision of this specific float64_tensor.
Any help would be great 
If you think that’s a good idea too, maybe I can ask for a freeze_dtype option for the register_buffer method as feature request.