Solved by googlebot in post #2 
              
              
                Try changing h to tensor list, e.g.: h = list(h.unbind()). Then cloning shouldn’t be needed. Not sure why you do f_h[idx].clone(), f_h is a tensor list, so I don’t see inplace operations with its elements, just element replacements. 
Inner (tau) loop looks optimizable/vectorizable. Matmuls done ther…