@ptrblck,
I fixed the NaN issue, the problem was while computing the statistics, before feeding it to the FC layer, I was computing the statistics across Time Dimension not across the feature dimension. And This was giving the NaN Values
Thank you so much @ptrblck for your feedback