So, the simpler workaround I found is to avoid registering my float64_tensor, and move it to the current device if needed during the forward pass. Plus I added hooks to the state_dict in order to correctly set or get this float64_tensor when saving or loading my module.
Snipped of code with workaround:
class custom_module(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.register_buffer('foo_tensor', get_foo_tensor())
        float64_tensor = get_float64_tensor()

        # Register state_dict hooks to get/set float64_tensor when saving/loading the module
        def set_f64_tensor_from_state_dict_hook(state_dict, *args, **kwargs):
                self.float64_tensor = state_dict['float64_tensor']
        self._register_load_state_dict_pre_hook(set_f64_tensor_from_state_dict_hook)
        def update_state_dict_hook(self, state_dict, *args, **kwargs):
                state_dict['float64_tensor'] = self.float64_tensor
                return state_dict
        self._register_state_dict_hook(update_state_dict_hook)

    def forward(self, x):
        input_dtype = x.dtype
        x = some_regular_op(x, self.foo_tensor)

        # Change device of f64 tensor if needed
        self.float64_tensor = self.float64_tensor.to(x.device)

        # Avoid any autocast context messing around here
        with torch.cuda.amp.autocast(enabled=False):
            x = x.to(dtype = torch.float64)
            x = some_high_precision_op(x, self.float64_tensor)

        x = x.to(input_dtype)
        return x

Which is a good enough solution to me. But please, don’t hesitate to comment if you see any problem with this approach or if you have any smarter move.