Hi, Jeremy!
Let me preface this by saying I’m practically a noob with regard to pytorch in general, and less than that for robotics so feel free to ignore my thoughts… 
My first thought: why are ML techniques required to infer the position of any part of the robot?  Isn’t this mathematically deterministic?
My second thought: granted my pytorch experience is limited, but I’m not sure why a custom function as a network layer wouldn’t work.  With that said, I’d need to see more of your code to give you solid feedback.
Good Luck!
SEH
EDIT: ahh, I think I get it… You’re using the net to give you the positions of your servos that will be required to meet each pose??