Since this question has been unanswered for a month, I will attempt to answer it. Glow developers/anyone else, please feel free to correct me wherever I go wrong.
If I understand your question correctly, you are asking if one can use Glow to generate code for a parallel hardware so as to distribute compute and data, something similar to distributed TensorFlow/PyTorch. Is that accurate?
If that is the case, you can model your system in OpenCL such that your CPU is the host and your parallel hardware — a GPU, a cluster of CPUs, or any accelerator, etc. — is the device (provided there are OpenCL drivers for that device). Then, you could layout your convolution kernel on the parallel device and do the softmax on your CPU (or another device). The OpenCL drivers will take care of copying the code and data. The memory model of your system (shared vs. distributed, for instance), communication links (PCIe vs. nvlink, etc.) will dictate the choice of your compute model, how your lay out code and data, the granularity level of your kernels, etc.
The CPU backend as it is today (and what I perceive its intent to be) is to support running the network on a (single-core) CPU (X86, ARM, etc.).