I teach that in my PyTorch internals training, if you’re near Munich and want to book a seat… 
But so the theory answer is any sequence of elementwise ops and the practical answer is anything that you see merged into fusion groups in myfn.graph_for(*my_inputs) . (Only is done on GPU by default.)
In addition to the blog post linked above 42, there is my blog post on optimizing LSTM backwards 41 and an old talk from me using this on a simple example IoU in detail 24.
Obviously a lot more is to be had from extending what can be fused TorchTVM 64 is a (highly experimental) approach to hook into TVM which is a (also experimental in my experience) framework that can optimize also reductions, which the JIT cannot .
Personally, I think a lot more could be had, but I’m not sure who makes that a top priority at the moment.
Best regards
Thomas