Having a operator level profile might help to narrow down if certain ops are causing the slowdown.
From the model code, seems like there are some 1x1 conv’s in the network, and my understanding is that the performance of these may not be as efficient on fbgemm.
cc @dskhudia in case anything else stands out that may be causing slower inference on fbgemm