Yeah, you have a point.
Have you tried to time the processing of the different sizes on one GPU? It would be interesting to see, what the speedup should be.
Don’t forget to synchronize the GPU calls before stopping the timer.