You are right, the whole system is the variation of model parallelism. We were also thinking about data parallelism across the interfaces but that does not work as in this case PyTorch puts all input data on GPU 0 before distributing it across parallel nodes. For our scenario, the input data is so large that it takes more than entire 32Gb of GPU 0 memory itself.