As the snippet below, script model actually get slower than average time 0.12sec in the first iteration in for loop as @driazati mentioned above . However, after the first iteration, the rest is ~ 0.12 sec/img and is still twice longer than pytorch model.
Put more information here: batch_size=1
pytorch 1.4, python 3.7
# pytorch model
for i in range(image_number):
    out1 = model(input) # forward time: 0.05 sec

#torchscript model
ScriptModel = torch.jit.script(model, input)
for i in range(image_number):
    out2 = ScriptModel(input) # forward time: 0.12 sec if i > 0, 3 sec if i==0