Thanks for your info. Really appreciate!
While, I know there are already some work to control the bidi- or uni-directional attention by using a matrix. like this: unilm/unilm-v1 at master · microsoft/unilm · GitHub 1
so I am wondering if we could also realise flexible attention by using this in GPT2.