additional global attended token in bigbird-roberta
Hello, based on my understanding of the bigbird style attention mechanism, the only tokens in the ITC construction supported in HuggingFace that are global are the first and last tokens. Is there an easy way to add an additional global token position that should always be attended to? For example, if I want to make sure the second token in the sequence is always globally attended to, where in the modeling_bigbird.py (or elsewhere) do I need to add that, or is there another easier way to pass additional globally attended token positions? Any help would be greatly appreciated. Thank you.
cc @vasudevgupta7 <|||||>Hey @calderma, sorry for late reply. I somehow missed your comment.

Yes, only ITC code is supported for now and hence only 1st & last **blocks** (i.e collection of tokens) are global. So, you can control which tokens should be global by playing around with block_size a bit, though this would increase compute a bit as random & sliding tokens are also dependent on it.

But in your case, as you want to make 2nd token as global, it will be a global token if block size is > 2 (Note: default block size is 64).<|||||>Ah ok got it. I thought it was just the first token, not the first block. My mistake. Thanks!