I think the reason may be,
Few reasons

Parameters updates are too large and its overshooting the gradient. The optimization process is unstable, it diverges instead of converging to a minimum.
Since weights and bias are at extreme end after first epoch, it continues to fluctuate causing loss to move to inf. Solution is to normalize the X to [-1, 1] or [0,1].
I was using SGD, which is sensitive to scaling and makes the parameters to overshoot. If i use Adam optimizer i.e. adaptive learning rate the model tends to find convergence even without scaling down the X to [-1, 1] or [0, 1].