Thank you for your response. I realized i needed to work on the optimization parameters. For example choosing between Least squared loss or cross entropy, ADAM optimizer seems to be quite better than SGD, and that helped me a lot.