The underlying workflow of DataParallel is described in this blog post 737 by @Thomas_Wolf in a detailed way.
You could use the functional API and apply the scatter and gather methods manually, if you need more control over what’s being executed.