After I trained it on some data the accuracy of the model dropped about 20% for the same classes that I want to make prediction on. I would expect it to be around 94% which was the case before training. So something is going on.