Break your model out of DataParallel, replace the layers, and then wrap it with DataParallel again. I’m assuming you won’t need to do this enough to cause any severe overhead.
I’m unsure on that warning. I’m assuming this is a separate problem altogether.