What do you mean by network capacity?
However, the problem is not over-fitting, that would be the next step. In order to do that I’m plotting training and testing accuracy at intermediate epochs. But the test accuracy at the last epoch changes depending on if I’m doing testing at intermediate epochs or not, which is strange.