I am sorry, I should have done that in the first place. This is the repository. https://github.com/amdegroot/ssd.pytorch.git 7. Here is another issue which has the same problem https://github.com/amdegroot/ssd.pytorch/issues/215 10
I have created a fork here to reproduce the bug: https://github.com/TheShadow29/ssd.pytorch 3.
I simply downloaded VOC2007, and VOC2012, extracted them and then create a symlink to the VOCdevkit in the folder data/ after setting up VOC dataset. I think there is should be an easier approach, but that would require a bit more changes to the code so I have not done that.
Then download pre-trained weight https://github.com/amdegroot/ssd.pytorch#download-a-pre-trained-network 4 from here and put it into the weights folder
Then in eval2.py I just do a forward pass using pre-trained weights and without loading any weights.
Here is my eval2.py code
if __name__ == '__main__':                                                                                                      
    # load net                                                                                                                  
    num_classes = len(labelmap) + 1                      # +1 for background                                                    
    net = build_ssd('test', 300, num_classes)            # initialize SSD                                                       
    net.cuda()                                                                                                                  
    net.eval()                                                                                                                  
    st_time = time.time()                                                                                                       
    o = net(torch.rand(1, 3, 300, 300))                                                                                         
    print(time.time() - st_time)                                                                                                
                                                                                                                                
    if args.trained_model:                                                                                                      
        print('Loading Existing Model', args.trained_model)                                                                     
        net.load_state_dict(torch.load(args.trained_model))                                                                     
                                                                                                                                
    net.cuda()                                                                                                                  
    net.eval()                                                                                                                  
    st_time = time.time()                                                                                                       
    o = net(torch.rand(1, 3, 300, 300))                                                                                         
    print(time.time() - st_time)                                                                                                                                                                                        

Then I run
python eval2.py --trained_model weights/ssd300_mAP_77.43_v2.pth
And this is the output:

Firs is the time for forward pass of the network without weights loaded, second is with weights loaded.
Let me know if there is something missing in the setup.