Thanks @ptrblck, instantiating model directly and calling model.to(device) take a similar amount of time?
Total time: 5.33244 s
Function: load_model at line 85

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    85                                           @profile
    86                                           def load_model(dirname, device, weights=None):
    94         1         17.0     17.0      0.0      weights = os.path.join(dirname, 'weights_%s.tar' % weights)
    95         1         81.0     81.0      0.0      device = torch.device(device)
    96                                           
    97                                               #modelfile = os.path.join(dirname, 'model.py')
    98                                               #model = torch.load(modelfile, map_location=device)
    99                                           
   100         1          8.0      8.0      0.0      config = os.path.join(dirname, 'config.toml')
   101         1     105144.0 105144.0      2.0      model = Model(toml.load(config))
   102         1    5154339.0 5154339.0     96.7      model.to(device)
   103                                           
   104         1      71250.0  71250.0      1.3      model.load_state_dict(torch.load(weights, map_location=device))
   105         1       1600.0   1600.0      0.0      model.eval()
   106         1          1.0      1.0      0.0      return model