Hey. I am working on testing diffrent methods of hyperparameter optimalization. First i want to test grid search, this method requries to train same type of network, using same dataset, same optimzer but setting diffrent parameters of this optimazer. Is it possible to train many models of networks at once. Currently I training single network, then I set diffrent paramaters and I am traning again, but this tooks a lot of time.
This is code of my current training process, where “myparam” is an array of diffrent parameters. There is 81 different combination so, the main loop is done 81 times.
for x in range(81):
    torch.cuda.empty_cache()
    model = models.alexnet(pretrained=True)
    model.classifier[6] = torch.nn.Linear(model.classifier[6].in_features, 9)
    device = torch.device('cuda')
    model = model.to(device)
    optimizer = optim.SGD(model.parameters(), lr = myparam[(x,0)], momentum=myparam[(x,1)])
    print("Attempt: ",x)
    
    train_loader = torch.utils.data.DataLoader(
        train_dataset,
        batch_size=10,
        shuffle=True,
        num_workers=2
    )

    test_loader = torch.utils.data.DataLoader(
        test_dataset,
        batch_size=10,
        shuffle=True,
        num_workers=2
    )
    for epoch in range(NUM_EPOCHS):
    
    
        for images, labels in iter(train_loader):
            images = images.to(device)
            labels = labels.to(device)
            optimizer.zero_grad()
            outputs = model(images)
            loss = F.cross_entropy(outputs, labels)
            loss.backward()
            optimizer.step()
    
        test_error_count = 0.0
        for images, labels in iter(test_loader):
            images = images.to(device)
            labels = labels.to(device)
            outputs = model(images)
            test_error_count += float(torch.sum(torch.abs(labels - outputs.argmax(1))))
    
        test_accuracy = 1.0 - float(test_error_count) / float(len(test_dataset))