Hi @apaszke,
Sure, here is my complete snippet:
 class Net(nn.Module):
  def __init__(self):
    super(Net, self).__init__()
         
    self.bn1 = nn.BatchNorm2d(8)
    self.bn2 = nn.BatchNorm2d(16)

    self.conv1 = nn.Conv2d(1, 8, 3 ,stride = (1,1), padding = (1,1))
    self.conv1 = torch.nn.DataParallel(self.conv1, device_ids = [1, 2])           
    self.conv2 = nn.Conv2d(8, 16 ,3, stride = (1,1), padding = (1,1))
    //etc ...

  def forward_prop(self, x):
 
    x = F.max_pool2d(F.relu(self.bn1(self.conv1(x))), (2,2))
    x = F.max_pool2d(F.relu(self.bn2(self.conv2(x))), (2,2))    

This is my only change. I will also mention that if I simply remove the torch.nn.DataParallel line in the above, my code runs and trains fine.
Thanks,