The main problem is that if you forward a single image at a time, you have a very sequential process where each step is a rather small operation to do. So you are stuck doing a bunch of small operations one after the other even though the total model is relatively big.