No, you only wrap a single module in DataParallel and it will parallelize the whole subtree. Not sure what the error is, but it’s likely a bug in your model. I can’t help much without the message and stack trace.