Add: Most surprising thing happens, it reaches up to 95% of training accuracy after 20 epochs when applied it on 50 samples randomly chosen from the main dataset of 5000 samples. But when I added more data in the training set, its training accuracy falls to almost 35% even after 500 epochs.
Please anyone suggest me what’s wrong?