@jerryzh168 I don’t see how replacing relu6 with relu doesn’t translate to a performance drop. It may work with MobileNetV2 but it isn’t a general approach.
Also, I don’t understand why Pytorch doesn’t offer ConvReLU2d similar modules for relu6 and hardswish. They are a common pattern for nets that runs on low end or IoT devices like smartphones. At least, a ConvReLU6_2d  for relu6 as it’s like a relu.
Moreover, I think it’s worth to add both patterns as like Regnet paper says  “We find that Swish outperforms ReLU at low flops, but ReLU is better at highflops.   Interestingly,  if g is  restricted  to  be  1  (depthwiseconv), Swish performs much better than ReLU. This sugggests that depthwise conv and Swish interact favorably, although the underlying reason is not at all clear.” So, for lower flops and quantized network you may want to use hardswish instead of relu as activation.