Very interesting.
Funny enough I was trying to implement a different summarization model- https://nlp.stanford.edu/pubs/see2017get.pdf 4
I had to severely limit the sequence length or model size to run it at max memory usage.
There is no-intra decoder attention in this model but memory usage is still pretty high. I’m convinced that the reason is storing gradients for all time steps. Which is to be expected.
But I got it to work and am training it right now which takes a long time.
If anyone is interested - https://github.com/hashbangCoder/Text-Summarization 14