We’re still working on that. It’s true that replicate can add some overhead for larger networks, and that’s why we also have a DataParallel module 27. The overhead of separate broadcasts isn’t very large from what we’ve seen. Nevertheless, if it’s quite high, we’re going to overlap the transfers with the computation, so the overhead is going to be even smaller if we were to sync the flattened parameters.
We don’t support flattening the parameters. It’s quite complex and bug prone to do that correctly, and maintain flexibility.
The main problem is that with 8 GPUs we’re a bit limited because of Python’s GIL (only one thread can execute Python code at a time). That’s why we’re working on moving more of logic commonly used in the vision networks, as well as some others, to our C backends, so they can proceed without blocking other threads.