The inference speed depends on a lot of factors, e.g.:

are you loading each image, pushing to the device, and running a single forward pass?
If so, are you adding the timing of all steps or just the actual forward pass?
Half precision might help, if you use a GPU with Volta/Tensor Cores
Yes, it looks like your model is overfitting on the training data, as the accuracy gap is quite large