On this topic, we are closely integrating with TVM now, with some engineering behind it. Please see https://github.com/pytorch/tvm 21
We are less inclined about NNVM – TVM provides Relay, which is a high-level IR.