I found out that the problem lies in the “Conv packed params hoisting” and “INSERT_FOLD_PREPACK_OPS” steps, whatever that means…
The following code optimizes the model for mobile without doubling the model’s size by blacklisting those optimization steps, this works without the nightly build:
import torch
from torch.utils.mobile_optimizer import optimize_for_mobile
from torch._C import MobileOptimizerType

torchscript_model = torch.jit.script(model)
torchscript_model = optimize_for_mobile(
    torchscript_model,
    optimization_blocklist={
            MobileOptimizerType.HOIST_CONV_PACKED_PARAMS,
            MobileOptimizerType.INSERT_FOLD_PREPACK_OPS
    },
)
if use_lite:
    torchscript_model._save_for_lite_interpreter("model.ptl")
else:
    torch.jit.save(torchscript_model, "model.pt")