There is a solution to get intermedia output by forward hooks using multi-gpus in this post 183 although not so elegant. It dose work in my test.
But the weird phenomenon explained in above post  still confuses me.