That the output is another sentence from the training data is very odd. Something garbled would just mean that the training fails somewhere. But you really get a proper sentences as output, just not the input one.
Did you try to iteratively build the model. Any Seq2Seq model for machine translation can be used as an autoencoder, only that input and target are the same sequence. So one can start with the basic PyTorch Seq2Seq 11 tutorial. If this is training fine, one can introduce additional bottelneck layers between the encoder and decoder to see how it’s affecting the training – or even extend it to a Variational Autoencoder.
Can you overtrain the model in a small training dataset?
Well, I had a quick look at your code, but it’s far from a minimal example :). Here are just some things I’ve noticed:


In the line hidden = hidden.view(self.n_layers, 1, input_seq.shape[1], -1)[-1], why is the second parameter a 1? This should be 2 since it represents the number of directions and you use a bidirectional GRU.


Why do you need the encoder output encoded in for the decoder? What is the decoder_input = torch.LongTensor([1 for i in range(encoded.shape[1])])... doing. Usually you have to initialize the target sequence with a special start token, e.g., input = torch.LongTensor([[Token.SOS]] * batch_size).to(self.device)


You may want to have a look at my code for an Autoencoder 7 and Variational Autoencoder 2. I started with the PyTorch Seq2Seq tutorial, extended it to the AE, and then to the VAE.