I’m unsure what this indexing sequence_output = outputs[0] is grabbing, but besides that the code looks alright (at least I don’t see any obvious issues).
Since you are using two output units in the last linear layer, I assume you are using nn.CrossEntropyLoss for the 2-class multi-class classification?
While this would work, you could also return a single output and use nn.BCEWithLogitsLoss instead, so you might want to experiment with it.
If you are running out of memory on the GPU you would indeed need to lower the memory usage by e.g. lowering the batch size. If it’s already a single sample, you could use e.g. torch.utils.checkpoint to trade compute for memory.