I think the two most important parameters to save memory are:

sequence length as all your hidden states (and their gradients) have that dimension
vocabulary size as it controls directly the size of the biggest weight matrix in your model (the embeddings table)