As you said, you can save the outputs of the FormerNet on disc (if it is not too much), and assuming the required time for IO is less than evaluating the FormerNet.
I have done something similar where I saved the outputs of a VGG network (which were tensors of size 2062) in « .h5 » format. This allowed me to save GPU memory and increase computation speed.