Bit more info from my debug:
The module where the distributions are generated from, if I put the forward pass of only that module with autocast False, then it works for 14 iterations and then throughs the exact same error.
Here is the line where the distribution is generated:-
dist = Independent(Normal(loc=mu, scale=torch.exp(log_sigma)),1)