Yes I think the problem is the size of the data set. Thanks for your suggestion I ll try on another standard dataset.

I just see above that you only have 2400 examples, which could be the main reason like you suggest.



 fermat97:

they are matrices which are very very sparse with the size 784x162. Almost all the values are zero except a few of them which have real values, could that be a reason?


Almost all values being 0 could be a problem, but it’s probably not the main reason. MNIST images also contain lots of 0’s. Another thing though is, besides the small dataset size, that 784x162 is very large for a convenet (typically, even for images, standard resnets for e.g,. face recognition operate on images between ~60x60 and ~200x200.
Since you are mentioning that these are not images, I wonder if it is a tabular dataset, in which case you might be better off using a network with only a few (e.g., 1-3) fully connected layers with dropout.