I had an issue 6 with a project of mine where a GRU failed to optimize and always predicted the same token. After a lot of debugging I found out that the input was of dtype float and the initial given state that I passed was a double. This caused the outputs and hidden states to explode, which in turn seemed to cause issues for the optimizer and eventually the model would always predict the same token.
My question is, why are different dtypes an issue? Why does it lead to an explosion in the GRU? If different dtypes are problematic, perhaps PyTorch should throw a warning?