Thank you so much for your great response! This is very useful.

In the application we are restricted to batch_size = 1. Feed-forward only, so as in your script, we turn off torch grads.
Synchronization was actually performed after the 3 layers, using a Sequential as in your code.
It definitely makes sense that for small inputs and kernel sizes the overheads prevent seeing the flop count speedup in runtime.

My only remaining question is: why is memory access dominant in the bottleneck case? True, each convolution layer has to read and write the output, but the inputs and outputs of the layers are smaller than the baseline. Isn’t the number of memory accesses proportional to the number of mult-adds, as we need to read each input when we are using it in the convolution, so shouldn’t the overall memory access always be proportional to the compute time, and therefore we should be seeing more speed-up (that depends on the convolution compute time only)?
Thank you so much again.