Are you just trying this shape (in_channels  = 3, out_channels=32, kernel_size=3, stride=2 and padding=1) or is this actually used in a network? The reason I ask this is because usually the first conv is in_channels  = 3, out_channels=64, kernel_size=7, stride=2 and padding=3 and we do have optimized fast path for this in fbgemm.