The first one u mentioned might be the reason why only two devices are triggered.
So should I set all the data.cuda() to data.to(device)?
[ERROR]
Variable ‘delta’ is in another GPU, which shows that the second GPU 
want to calculate something with ‘delta’, but the variable is in 
another GPU device

I simplify my project as follow to describe the error above:
class model_A(nn.Module):
...
   def forward(self, input):
      delta = ......
      delta = delta.cuda()

class model_B(nn.Module):
...

class model_C(nn.Module):
   def __init__(self, .....):
      self.M_A = model_A(...)
      self.M_B = model_B(...)
   def forward(self, input):
      A,B,C = self.M_A(input)
      D = self.M_A(A, B)
      ...