@ptrblck for one other scenario, I am trying to test out one network with varying batch size, learning rate, momentum and more importantly network initialization method. I am trying it out like this
best_net = Net(conv1_kernel_size=3, conv1_output_channels=24, conv2_kernel_size=5, conv2_output_channels=24, fc1_output_size=200)
for bs in batch_size:
    trainloader = torch.utils.data.DataLoader(trainset, batch_size = bs, shuffle = True, num_workers = 2)
    testloader = torch.utils.data.DataLoader(testset, batch_size = bs, shuffle = True, num_workers = 2)
    for init in inits:
        apply_init(best_net, init)
        init_dict = best_net.state_dict()
        for lr in lrs:
            best_net.load_state_dict(init_dict)
            # optimizer.reset()
            optimizer = optim.SGD(best_net.parameters(), lr = lr, momentum = momentum)

Where apply init will just call the following function
def init_weight_xu(m):
    classname = m.__class__.__name__
    if classname.find('Conv') != -1 or classname.find('Linear') != -1:
        torch.nn.init.xavier_uniform_(m.weight) 
        m.bias.data.fill_(0.1)

The first run went well starting at 84% test acc and going till 91% in 15 epochs. But in 2nd run, with new lr, it started with 17%. I doubt that something is not getting updated. Either the network is not getting reset or the optimizer.
Can you please help me with this?