Yep, I think there is quite a bit of glue logic that can be encapsulated away. I may give it some thought later, once I have more experience with glow.
A bit off topic: my impression is that currently there is really only the cpu backend; is this correct? In that case, does Glow perform optimizations like taking advantage of CPU SIMD architecture? In particular quantization would make the network amenable to that sort of optimization it seems. If so, can we verify those optimizations by dumping low level IR or even LLVM asm output (the latter would probably be more appropriate in this case).