@ptrblck, @SandPhoenix
Update from my end.
The input x had a NAN value in it, which was the root cause of the problem.
This NAN was not present in the input as I had double checked it, but got introduced during the Normalization process.
Right now, I have figured out the input causing this NAN and removed it input dataset.
Things are working now.
@SandPhoenix: do please check your input for NAN’s before passing it to the Network Layers.