Solved by rad in post #2 
              
              
                Hey! 
PyTorch provides the Einsum function, which allows you to do different tensor operations based on indices. See here, here, here for more explanations. I find einsum the easiest way to do such operations. 
(Also, I would assume that matrix is actually of shape [batch, hidden_in, hidden_out], as…