I think the FairSeq code doesn’t work in your case, as your input shape seems to be static and you already run into OOM issues using batch_size=1.
What GPU are you using, i.e. how much memory is available?
You might want to reduce the number of filters to reduce the memory footprint.
Also, torch.utils.checkpoint 71 might be another approach to trade compute for memory.