Its normal for the first few layers to dominate the time, since the image is really large at the start, hasnt gone through any maxpooling layers yet.
if you can find a solution to this, I guess this is paper-worthy