Hi @jasonhuh,
This may have something to do with the shape of the input used in the Conv layers. My guess is that if the shape is not big enough then the work per thread is not significant enough for us to see the wins.
@dskhudia do you have any additional insights here about how FBGEMM may be behaving differently for single vs multi-thread here?