Hi,
I’m training a simple autoencoder over several GPUs (probably 4) with a batch size of 256 to 512. I have millions of examples to train. I want to make sure I am doing the right thing.
Below I have defined the autoencoder where I add self.encoder= nn.DataParallel(self.encoder) . Please see below:
class AutoEncoder(nn.Module):
    def __init__(self, n_embedded):
        super(AutoEncoder, self).__init__()
        self.encoder = nn.Sequential(
            nn.Linear(6144, n_embedded))
        self.encoder= nn.DataParallel(self.encoder)
        self.decoder = nn.Sequential(nn.Linear(n_embedded, 6144))
       
    def forward(self, x):
        encoded = self.encoder(x)
        decoded = self.decoder(encoded)
        return encoded, decoded

I initiate my model before training by:
model = AutoEncoder(2048)
model= nn.DataParallel(model)
model.to(device)
criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(),weight_decay=1e-5)


Would this get the most out of the GPUS?