Solved by iAbhyuday in post #6 
              
              
                [UPDATE] 
Got better results after switching to SGD from Adam Optimizer. But validation accuracy and loss saturate early . 
Learning curves (before early stopping) 
[train]