Sorry for forgetting the second part of the right approach. 
The new error is not raised by input.clone(), but should still be raised by self.latest_input, which is now on the CPU as it wasn’t properly registered.
To make sure model.to() pushes all internal parameters and buffers to the desired device you have to register self.latest_input as an nn.Parameter, if it should be trained, or as a buffer if it shouldn’t:
self.latest_input = nn.Parameter(torch.zeros(channel, width, height)
# or
self.register_buffer('latest_input', torch.zeros(channel, width, height))