The problem was actually with using torch.bmm in order to make diagonal covariance matrix, to be precise it wasn’t working the way I assumed it to worked so instead of using torch.bmm, I used torch.diag_embed()