Language Model, Transformers, Deep Learning, Artificial Intelligence, Machine Learning.
considering the global context and maintaining efficiency, as dilated attention serves as an efficient approximation of the dense attention matrix. Two more tricks LONGNET employs for better modeling. It incorporates varying dilation rates in each attention head for more diversity. It also