You are missing some critical issue in your implementation: the dimension of the conv layers input is changing through the model depth.
So i guess the right way to compute number of flops will be by using forward hook.