I was wondering if there is a way to train multiple independent neural networks in parallel on CPU only with random grid search?
Imagine a trivial example of a network like stated below (this is just an example):
import torch.nn as nn
import torch

class NeuralNet(nn.Module):
    def __init__(self):
        super(NeuralNet, self).__init__()
        self.Linear = nn.Linear(1, 1, bias=True)

    def forward(self, x):
        out = self.Linear(x)
        return out

I subsequently perform some form of grid search:
from skorch import NeuralNetRegressor
from skorch.callbacks import EarlyStopping
from sklearn.model_selection import RandomizedSearchCV

es = EarlyStopping(monitor='valid_loss', patience=5, threshold=1e-4, threshold_mode='abs', lower_is_better=True)

model = NeuralNetRegressor(
        module= NeuralNet,
        criterion=nn.MSELoss,
        optimizer=torch.optim.SGD,
        optimizer__momentum=0.95,
        optimizer__weight_decay=0.2,
        max_epochs=200,
        batch_size=32,
        lr=1e-4,
        callbacks=[es]
    )
model.verbose = 0
params_grid = {
        'lr': [1e-5, 1e-4],
        'optimizer__momentum': [0.9, 0.95, 0.99],
        'optimizer__weight_decay': [0.1, 0.9, 1.5]
}

gs = RandomizedSearchCV(model, params_grid, refit=True, cv=3,
n_jobs=5, n_iter=10, random_state=1234, scoring='neg_mean_squared_error')
gs.fit(x_train, y_train)


Can I perform the above simultaneously with let’s say three different models all at the same time, where all models are completely independent? i.e. can I perform three independent grid searches for three independent models at the same time if I have let’s say 80 cores or so and each grid search uses 20 cores.
My approach was to define a function that performs the grid search, and then execute the function in parallel but it did not seem to speed up any of my calculations? How would you approach a problem like this?
I hope this is not a stupid question!