Sorry for the long reply, but I rather reply in detail.
Some code is indeed missing (see the first comment in step()) because I did not want to bloat the code too much with boiler plate. However, concerning your comment, decoder_input is different at every iteration: it is set to the highest prediction of the previous output in the very last line of the for-loop:
decoder_input = topi.detach()

This is a typical scenario in e.g. machine translation or autoregressive language modelling where you predict a token based on the previous best output. (Some information is missing here, particularly the initial hidden state decoder_hidden, but I don’t think it’s important for the problem I am having.)
So to recapitulate: my problem is that I am running out of memory when using a large model (50k+ embedding size), even with small batch sizes but I believe that this should be able to be remedied by another gradient accumulation set-up. I am just not sure what the normal way-to-go is in an RNN set up where you do multiple forward passes per batch, and this accumulate a lot of history.
Usually you do gradient accumulation on the batch level (accumulate n batches and only then do the optimizer step), whereas here it seems to me to make more sense to this on the sequence level (accumulate n predictions in the for i in range(target_len) loop). I am just not 100% sure whether that makes sense.
I am right in assuming that in this code snippet:
loss += (item_loss / gradient_accumulation_steps)

loss will accumulate all the history of all iterations, right? So if target_len=200, loss will contain the history of 200 forward passes? That’s why I think that the gradient accumulation and optimizer should perhaps be moved to inside that for i in range(target_len) loop.
I think the right way to go is to call backward in the same loop as the model’s forward. The question then becomes how it should be normalised. IAssuming that target_len is always 200, if I call optimizer step in the same spot as I do know, that means that the gradients have been accumulatedgradient_accumulation_steps  *  200 times every time optimizer.step is called. If that is the case, then the loss should probably always be normalized by that amount, right? So step would look like this
    def step():
        """Processes one step (one batch) by forwarding multiple times to get a final prediction for a given sequence."""
        # do stuff... init hidden state and first input etc.
        loss = 0.
        
        for i in range(target_len):
            with torch.cuda.amp.autocast(enabled=amp):
                # overwrite previous decoder_hidden
                output, decoder_hidden = model(decoder_input, decoder_hidden)
                # Not sure why I need to detach here
                decoder_hidden = decoder_hidden.detach()

                step_loss = self.criterion(output, target_tensor[i]) / (self.gradient_accumulation_steps * target_len)
                
                loss += step_loss.item()

            if do == "train":
                if self.amp:
                    # for ANP see https://pytorch.org/docs/stable/notes/amp_examples.html
                    self.scaler.scale(step_loss).backward()
                else:
                    step_loss.backward()

            topv, topi = output.topk(1)
            decoder_input = topi.detach()
        
        return loss.item() / target_len

After trying this, I got the “Trying to do a second backward pass” so I hadd to add                 decoder_hidden = decoder_hidden.detach() although I am not entirely sure why.