<p dir="auto">Hi,<br>
I have a requirement that model should search for relevant documents before answering the query. I found RAG from Facebook AI which perfectly aligns with my requirement . Also saw <a href="https://huggingface.co/blog/ray-rag" rel="nofollow">this</a> blog from HF and came to know that HF implemented RAG which is awesome!</p>
<p dir="auto">My doubt is whether I could extend HF's implementation of RAG so that the model should retrieve documents from local directory rather than HF's wikipedia corpus.<br>
Are there any notebooks to refer..?</p>
<h1 dir="auto"><g-emoji class="g-emoji" alias="rocket" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f680.png">🚀</g-emoji> Feature request</h1>
<p dir="auto"><strong>Good Second Issue</strong> - A more advanced issue for contributors who want to dive more into Longformer's attention mechanism.</p>
<p dir="auto">Longformer currently only outputs global attentions, which is suboptimal because users might be interested in the local attentions as well. I propose to change the "output_attention" logic as follows in longformer:</p>
<p dir="auto"><code class="notranslate">attentions</code> should correspond to the "local" attentions and then we'll add a new output type <code class="notranslate">global_attention</code> that contains the global_attentions. This is consistent with the naming of <code class="notranslate">attention_mask</code> and <code class="notranslate">global_attention_mask</code> IMO and the cleanest way to implement the feature.</p>
<p dir="auto">Implementing this feature would mean to that Longformer will require its own <code class="notranslate">ModelOutput</code> class =&gt;<br>
<code class="notranslate">BaseModelOutput,</code> =&gt; <code class="notranslate">LongformerBaseModelOutput</code> or <code class="notranslate">BaseModelOutputWithGlobalAttention</code> (prefer the first name though)<br>
<code class="notranslate">BaseModelOutputWithPooling,</code> =&gt; ...</p>
<p dir="auto">Also some tests will have to be adapted.</p>
<p dir="auto">This is a slightly more difficult issue, so I'm happy to help on it. One should understand the difference between local and global attention and how Longformer's attention is different to <em>e.g.</em> Bert's attention in general.</p>
<p dir="auto">For more detail check out discussion here: <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="654458686" data-permission-text="Title is private" data-url="https://github.com/huggingface/transformers/issues/5646" data-hovercard-type="issue" data-hovercard-url="/huggingface/transformers/issues/5646/hovercard" href="https://github.com/huggingface/transformers/issues/5646">#5646</a></p>