GPT2 attention mask
I want to use gpt2 to generate a list of options

> an option is a sentence starting with a special token '<option>'.

As I don't want the following options to rely on the previous options, I think I should mask all the previous options.

I could simply implement that during generation by generating one option per time, but I don't know how to do that during training.
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Thanks!<|||||>Thanks for reminding! 
https://discuss.huggingface.co/t/dynamic-attention-mask-during-gpt-2-training/2789
On 12/11/2020 06:16，Lysandre Debut<notifications@github.com> wrote：

Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the forum instead?

Thanks!

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.