Solved by G.M in post #5 
              
              
                @JonathanSum is correct. Increasing the number of neurons/layers don’t necessarily enhance the performance. In fact, too many neurons can cause over-fitting on small amounts of data. Increasing the layers can cause problems, too(u can read the ResNet paper). But for ur case, I think the bad performa…