Ok. So, I found out the problem. It is the way I have been transforming the parameters at the init step of the custom layer. using the registered log_alpha, log_beta, logsigmasq params directly instead of exponentiating them in the init step solved the issue.