Amazing.  This was my problem.  nn.ModuleList doesn’t seem to work when using nn.DataParallel as some GPUs have the weights in CPU mode and some have it on GPU mode.  What a hacky way to do it though but it seems we have no choice.  Thanks so much!