layer fusion is going to fuse Conv+BN into a Conv module or Conv + BN + ReLU into a ConvRelu module. this does not change numerics itself. Without fusion conv, bn and relu will be quantized independently, that might be the reason why the accuracy drops.