UPD:
I’ve tried to convert traced graph to Caffe2. And have run into same problem.
Traced with torch.jit.script model cannot be converted to Caffe2 because of  batchnorm.
model = torch.jit.load(buf)
f = io.BytesIO()
torch.onnx.export(model, x, f, example_outputs=outputs,
                  operator_export_type=torch.onnx.OperatorExportTypes.ONNX_ATEN_FALLBACK)
f.seek(0)

Where the model is traced backbone
~/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/onnx/utils.py in _optimize_graph(graph, operator_export_type, _disable_torch_constant_prop, fixed_batch_size, params_dict)
    157             torch.onnx.symbolic_helper._quantized_ops.clear()
    158             # Unpack quantized weights for conv and linear ops and insert into graph.
--> 159             torch._C._jit_pass_onnx_unpack_quantized_weights(graph, params_dict)
    160 
    161             # Insert permutes before and after each conv op to ensure correct order.

RuntimeError: false INTERNAL ASSERT FAILED at /opt/conda/conda-bld/pytorch_1586761698468/work/torch/csrc/jit/passes/onnx/unpack_quantized_weights.cpp:99, please report a bug to PyTorch. Unrecognized quantized operator while trying to compute q_scale for operator quantized::batch_norm