Solved by Haxxardoux in post #2 
              
              
                Update: The SHA-RNN codebase, which claimed to be very memory and computationally efficient, just uses torch.cat. I have been decieved.