I would try a wider range of learning rates. Fit the model with 0.1, 0.01, 0.001, 0.0001 and see what sorts of outcomes that you get. Is the training accuracy comparable to the testing accuracy? Also, what is the prior for each of your training classes? Are they things that might be hard to distinguish? Are they things similar to the classes in imagenet? These are all questions that might impact how well training works in your model.