agemagician:

it is better to use “BigBird” model than “RoBERTA”


Do the attention patterns that Big Bird uses have any biological sense or could they deteriorate performance for some tasks?
I mean, BigBird induces sparsity to the model using different attention patterns (see figure below) that have a certain linguistic sense, however, I don’t know if they can be easily extrapolated to relationships between k-mers.

image1506×414 25.1 KB