Andy-jqa:

nn.DataParallel


Hi, Andy-jpa
If the the first gpu (id:0) is occupied  all the time and no more space to use, you could consider to use only 1 and 2 gpu explictly assigned in you code by device_ids.
refer : https://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html 34
code like this:
def data_parallel(module, input, device_ids, output_device=None):
    if not device_ids:
        return module(input)

    if output_device is None:
        output_device = device_ids[0]

    replicas = nn.parallel.replicate(module, device_ids)
    inputs = nn.parallel.scatter(input, device_ids)
    replicas = replicas[:len(inputs)]
    outputs = nn.parallel.parallel_apply(replicas, inputs)
    return nn.parallel.gather(outputs, output_device)