I noticed the problem. And I have modified my code.
Now my code is as blow,
model = nn.DataParallel(model).cuda()
...
  for i, (input, target, num_frames) in enumerate(train_loader):
        target = target.cuda(async=True)  
        ...
        # do some processing on the input 
        output = model(input.cuda(), seg_num_list)

Is that right？Actually I noticed that  sending input without cuda() to the model can work well when using   DataParallel. I guess DataPerallel move input to GPUs automatically Inside its interior ？