Solved by Dudly01 in post #4 
              
              
                The solution was to zero out a certain index of the hidden states by multiplying those with zero and not by replacing it with torch.zeros()