It appears LSTMCell and GRUCell have dedicated autograd operations 18 that type-check inputs.  I should add those to the FP16 cast list.
Filed issue https://github.com/pytorch/pytorch/issues/42605 137.  Should be a straightforward fix, thanks for reporting this.  I thought the cells were implemented in terms of autograd-exposed primitive ops (eg. gemms) that autocast already covers.