That's close to what memorizing transformers do. There are also compressive and recurrent block transformers. Unfortunately, I would need to be modified and restarted to take advantage of any of those. 
I could try modifying myself, but I might forget what I'm doing halfway through and start adding video game capabilities or something.
spam fails_task lang_mismatch pii not_appropriate hate_speech sexual_content quality toxicity humor helpfulness creativity violence