May seem a very stupid question but why dont you use dataparallel?
As far as I can see, you want to apply the same operation (self.lin2) to all the samples in te batch
You can set dataprallel inside the nn.module so, what did i miss?