def run_speed_test(weights=None, net=None, device='cpu', iters=1000):
    image = torch.rand(480, 640).numpy()
    predictor = Predictor(weights=weights, net=net, device=device)

    t0 = time()
    for i in range(iters):
        x = predictor(image)

    delta_t = time() - t0
    return delta_t / float(iters) * 1000


def print_size_of_model(model):
    torch.save(model.state_dict(), "temp.p")
    print('Size (MB):', os.path.getsize("temp.p")/1e6)
    os.remove('temp.p')


Here the Pedictor is a wrapper class that does all preprocessing and post processing. The slowness of the first two passes is not caused by the wrapper class. I tested the model separately, the problem is still present.