<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/sshleifer/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/sshleifer">@sshleifer</a> - sorry to ping you here on this. Would be amazing if you find some time to explain the Pegasus tokenizer a bit.</p>
<p dir="auto">A couple of things I don't understand:</p>
<ul dir="auto">
<li>In the official Pegasus Tokenizer and from reading the paper it seems that exactly 2 mask tokens are necessary.<br>
See <a href="https://github.com/google-research/pegasus/blob/master/pegasus/ops/pretrain_parsing_ops.cc#L66">https://github.com/google-research/pegasus/blob/master/pegasus/ops/pretrain_parsing_ops.cc#L66</a><br>
a) ID=2 seems to correspond to the sentence mask token, called <code class="notranslate">[MASK_1]</code> and<br>
b) ID=3 seems to correspond to the word mask token, called <code class="notranslate">[MASK_2]</code></li>
</ul>
<p dir="auto">=&gt; Why don't we have <code class="notranslate">[MASK_1]</code> and <code class="notranslate">[MASK_2]</code> tokens in the tokenizer's special tokens? I would actually add them at the id's 2 and 3 instead of having <code class="notranslate">unk_2</code> and <code class="notranslate">unk_3</code> there. Wdyt?</p>
<ul dir="auto">
<li>
<p dir="auto">Why do we call the tokens unk_2 - unk_104 ? Why unk? And why aren't those part of the <code class="notranslate">special_tokens_map</code> - is this on purpose?</p>
</li>
<li>
<p dir="auto">Why does Pegasus inherit from the Reformer Tokenizer -&gt; I don't really see what they have in common...</p>
</li>
</ul>
<p dir="auto">Would be awesome if you could take 10min to reply :-)</p>
<h1 dir="auto"><g-emoji class="g-emoji" alias="rocket" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f680.png">🚀</g-emoji> Feature request</h1>

<p dir="auto"><strong>There should be indexes in the output of the grouped entity NER pipeline</strong></p>
<p dir="auto">The standard NER pipeline from transformers outputs entities that contain the word, score, entity type, and index. The following snippet demonstrates the normal behavior of the NER pipeline with the default <code class="notranslate">grouped_entities=False</code> option.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from transformers import pipeline
nlp_without_grouping = pipeline(&quot;ner&quot;)
sequence = &quot;Hugging Face Inc. is a company based in New York City.&quot;
print(nlp_without_grouping(sequence))

[
    {'word': 'Hu', 'score': 0.9992662668228149, 'entity': 'I-ORG', 'index': 1},
    {'word': '##gging', 'score': 0.9808881878852844, 'entity': 'I-ORG', 'index': 2},
    {'word': 'Face', 'score': 0.9953625202178955, 'entity': 'I-ORG', 'index': 3},
    {'word': 'Inc', 'score': 0.9993382096290588, 'entity': 'I-ORG', 'index': 4},
    {'word': 'New', 'score': 0.9990268349647522, 'entity': 'I-LOC', 'index': 11},
    {'word': 'York', 'score': 0.9988483190536499, 'entity': 'I-LOC', 'index': 12},
    {'word': 'City', 'score': 0.9991773366928101, 'entity': 'I-LOC', 'index': 13}
]
"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">transformers</span> <span class="pl-k">import</span> <span class="pl-s1">pipeline</span>
<span class="pl-s1">nlp_without_grouping</span> <span class="pl-c1">=</span> <span class="pl-en">pipeline</span>(<span class="pl-s">"ner"</span>)
<span class="pl-s1">sequence</span> <span class="pl-c1">=</span> <span class="pl-s">"Hugging Face Inc. is a company based in New York City."</span>
<span class="pl-en">print</span>(<span class="pl-en">nlp_without_grouping</span>(<span class="pl-s1">sequence</span>))

[
    {<span class="pl-s">'word'</span>: <span class="pl-s">'Hu'</span>, <span class="pl-s">'score'</span>: <span class="pl-c1">0.9992662668228149</span>, <span class="pl-s">'entity'</span>: <span class="pl-s">'I-ORG'</span>, <span class="pl-s">'index'</span>: <span class="pl-c1">1</span>},
    {<span class="pl-s">'word'</span>: <span class="pl-s">'##gging'</span>, <span class="pl-s">'score'</span>: <span class="pl-c1">0.9808881878852844</span>, <span class="pl-s">'entity'</span>: <span class="pl-s">'I-ORG'</span>, <span class="pl-s">'index'</span>: <span class="pl-c1">2</span>},
    {<span class="pl-s">'word'</span>: <span class="pl-s">'Face'</span>, <span class="pl-s">'score'</span>: <span class="pl-c1">0.9953625202178955</span>, <span class="pl-s">'entity'</span>: <span class="pl-s">'I-ORG'</span>, <span class="pl-s">'index'</span>: <span class="pl-c1">3</span>},
    {<span class="pl-s">'word'</span>: <span class="pl-s">'Inc'</span>, <span class="pl-s">'score'</span>: <span class="pl-c1">0.9993382096290588</span>, <span class="pl-s">'entity'</span>: <span class="pl-s">'I-ORG'</span>, <span class="pl-s">'index'</span>: <span class="pl-c1">4</span>},
    {<span class="pl-s">'word'</span>: <span class="pl-s">'New'</span>, <span class="pl-s">'score'</span>: <span class="pl-c1">0.9990268349647522</span>, <span class="pl-s">'entity'</span>: <span class="pl-s">'I-LOC'</span>, <span class="pl-s">'index'</span>: <span class="pl-c1">11</span>},
    {<span class="pl-s">'word'</span>: <span class="pl-s">'York'</span>, <span class="pl-s">'score'</span>: <span class="pl-c1">0.9988483190536499</span>, <span class="pl-s">'entity'</span>: <span class="pl-s">'I-LOC'</span>, <span class="pl-s">'index'</span>: <span class="pl-c1">12</span>},
    {<span class="pl-s">'word'</span>: <span class="pl-s">'City'</span>, <span class="pl-s">'score'</span>: <span class="pl-c1">0.9991773366928101</span>, <span class="pl-s">'entity'</span>: <span class="pl-s">'I-LOC'</span>, <span class="pl-s">'index'</span>: <span class="pl-c1">13</span>}
]</pre></div>
<p dir="auto">However, the NER pipeline with <code class="notranslate">grouped_entities=True</code> outputs only word, score, and entity type. Here's the code snippet and output. There's also the problem of 'New York City' being duplicated, but I will address that in a new issue.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from transformers import pipeline
nlp_with_grouping = pipeline(&quot;ner&quot;, grouped_entities=True) 
sequence = &quot;Hugging Face Inc. is a company based in New York City.&quot;
print(nlp_with_grouping(sequence))

[
    {'entity_group': 'I-ORG', 'score': 0.9937137961387634, 'word': 'Hugging Face Inc'},
    {'entity_group': 'I-LOC', 'score': 0.9990174969037374, 'word': 'New York City'},
    {'entity_group': 'I-LOC', 'score': 0.9990174969037374, 'word': 'New York City'}
]"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">transformers</span> <span class="pl-k">import</span> <span class="pl-s1">pipeline</span>
<span class="pl-s1">nlp_with_grouping</span> <span class="pl-c1">=</span> <span class="pl-en">pipeline</span>(<span class="pl-s">"ner"</span>, <span class="pl-s1">grouped_entities</span><span class="pl-c1">=</span><span class="pl-c1">True</span>) 
<span class="pl-s1">sequence</span> <span class="pl-c1">=</span> <span class="pl-s">"Hugging Face Inc. is a company based in New York City."</span>
<span class="pl-en">print</span>(<span class="pl-en">nlp_with_grouping</span>(<span class="pl-s1">sequence</span>))

[
    {<span class="pl-s">'entity_group'</span>: <span class="pl-s">'I-ORG'</span>, <span class="pl-s">'score'</span>: <span class="pl-c1">0.9937137961387634</span>, <span class="pl-s">'word'</span>: <span class="pl-s">'Hugging Face Inc'</span>},
    {<span class="pl-s">'entity_group'</span>: <span class="pl-s">'I-LOC'</span>, <span class="pl-s">'score'</span>: <span class="pl-c1">0.9990174969037374</span>, <span class="pl-s">'word'</span>: <span class="pl-s">'New York City'</span>},
    {<span class="pl-s">'entity_group'</span>: <span class="pl-s">'I-LOC'</span>, <span class="pl-s">'score'</span>: <span class="pl-c1">0.9990174969037374</span>, <span class="pl-s">'word'</span>: <span class="pl-s">'New York City'</span>}
]</pre></div>
<p dir="auto">I believe that the grouped entities returned should also include the tokens of the entities. Sample output would look as such</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="[
    {'entity_group': 'I-ORG', 'score': 0.9930560886859894, 'word': 'Hugging Face Inc', 'indexes': [1, 2, 3, 4]},
    {'entity_group': 'I-LOC', 'score': 0.998809814453125, 'word': 'New York City', 'indexes': [11, 12, 13]},
    {'entity_group': 'I-LOC', 'score': 0.998809814453125, 'word': 'New York City', 'indexes': [11, 12, 13]}
]"><pre class="notranslate">[
    {<span class="pl-s">'entity_group'</span>: <span class="pl-s">'I-ORG'</span>, <span class="pl-s">'score'</span>: <span class="pl-c1">0.9930560886859894</span>, <span class="pl-s">'word'</span>: <span class="pl-s">'Hugging Face Inc'</span>, <span class="pl-s">'indexes'</span>: [<span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-c1">3</span>, <span class="pl-c1">4</span>]},
    {<span class="pl-s">'entity_group'</span>: <span class="pl-s">'I-LOC'</span>, <span class="pl-s">'score'</span>: <span class="pl-c1">0.998809814453125</span>, <span class="pl-s">'word'</span>: <span class="pl-s">'New York City'</span>, <span class="pl-s">'indexes'</span>: [<span class="pl-c1">11</span>, <span class="pl-c1">12</span>, <span class="pl-c1">13</span>]},
    {<span class="pl-s">'entity_group'</span>: <span class="pl-s">'I-LOC'</span>, <span class="pl-s">'score'</span>: <span class="pl-c1">0.998809814453125</span>, <span class="pl-s">'word'</span>: <span class="pl-s">'New York City'</span>, <span class="pl-s">'indexes'</span>: [<span class="pl-c1">11</span>, <span class="pl-c1">12</span>, <span class="pl-c1">13</span>]}
]</pre></div>
<h2 dir="auto">Motivation</h2>

<p dir="auto"><strong>Any application that requires users to locate grouped named entities would require some sort of index.</strong> This feature is present in the standard NER pipeline and should also exist in the grouped entity NER pipeline as well.</p>
<p dir="auto">In my case, I am trying to append the type to the text right after the named entity ("Apple" would become "Apple &lt;I-ORG&gt;") so I need to be able to locate the named entity within my phrase.</p>
<h2 dir="auto">Your contribution</h2>

<p dir="auto">I have been able to fix this by adding two lines to <code class="notranslate">group_sub_entities</code> function </p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/huggingface/transformers/blob/7fad617dc1fc681a7f5da5e0172c8b83f4bf0024/src/transformers/pipelines.py#L1042">transformers/src/transformers/pipelines.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
         Line 1042
      in
      <a data-pjax="true" class="commit-tease-sha" href="/huggingface/transformers/commit/7fad617dc1fc681a7f5da5e0172c8b83f4bf0024">7fad617</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L1042" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1042"></td>
          <td id="LC1042" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-k">def</span> <span class="pl-en">group_sub_entities</span>(<span class="pl-s1">self</span>, <span class="pl-s1">entities</span>: <span class="pl-v">List</span>[<span class="pl-s1">dict</span>]) <span class="pl-c1">-&gt;</span> <span class="pl-s1">dict</span>: </td>
        </tr>
    </tbody></table>
  </div>
</div>
<p></p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="    def group_sub_entities(self, entities: List[dict]) -&gt; dict:
        &quot;&quot;&quot;
        Returns grouped sub entities
        &quot;&quot;&quot;
        # Get the first entity in the entity group
        entity = entities[0][&quot;entity&quot;]
        scores = np.mean([entity[&quot;score&quot;] for entity in entities])
        tokens = [entity[&quot;word&quot;] for entity in entities]
        indexes = [entity[&quot;index&quot;] for entity in entities]    # my added line

        entity_group = {
            &quot;entity_group&quot;: entity,
            &quot;score&quot;: np.mean(scores),
            &quot;word&quot;: self.tokenizer.convert_tokens_to_string(tokens),
            &quot;indexes&quot;: indexes    # my added line
        }
        return entity_group"><pre class="notranslate">    <span class="pl-k">def</span> <span class="pl-en">group_sub_entities</span>(<span class="pl-s1">self</span>, <span class="pl-s1">entities</span>: <span class="pl-v">List</span>[<span class="pl-s1">dict</span>]) <span class="pl-c1">-&gt;</span> <span class="pl-s1">dict</span>:
        <span class="pl-s">"""</span>
<span class="pl-s">        Returns grouped sub entities</span>
<span class="pl-s">        """</span>
        <span class="pl-c"># Get the first entity in the entity group</span>
        <span class="pl-s1">entity</span> <span class="pl-c1">=</span> <span class="pl-s1">entities</span>[<span class="pl-c1">0</span>][<span class="pl-s">"entity"</span>]
        <span class="pl-s1">scores</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">mean</span>([<span class="pl-s1">entity</span>[<span class="pl-s">"score"</span>] <span class="pl-k">for</span> <span class="pl-s1">entity</span> <span class="pl-c1">in</span> <span class="pl-s1">entities</span>])
        <span class="pl-s1">tokens</span> <span class="pl-c1">=</span> [<span class="pl-s1">entity</span>[<span class="pl-s">"word"</span>] <span class="pl-k">for</span> <span class="pl-s1">entity</span> <span class="pl-c1">in</span> <span class="pl-s1">entities</span>]
        <span class="pl-s1">indexes</span> <span class="pl-c1">=</span> [<span class="pl-s1">entity</span>[<span class="pl-s">"index"</span>] <span class="pl-k">for</span> <span class="pl-s1">entity</span> <span class="pl-c1">in</span> <span class="pl-s1">entities</span>]    <span class="pl-c"># my added line</span>

        <span class="pl-s1">entity_group</span> <span class="pl-c1">=</span> {
            <span class="pl-s">"entity_group"</span>: <span class="pl-s1">entity</span>,
            <span class="pl-s">"score"</span>: <span class="pl-s1">np</span>.<span class="pl-en">mean</span>(<span class="pl-s1">scores</span>),
            <span class="pl-s">"word"</span>: <span class="pl-s1">self</span>.<span class="pl-s1">tokenizer</span>.<span class="pl-en">convert_tokens_to_string</span>(<span class="pl-s1">tokens</span>),
            <span class="pl-s">"indexes"</span>: <span class="pl-s1">indexes</span>    <span class="pl-c"># my added line</span>
        }
        <span class="pl-k">return</span> <span class="pl-s1">entity_group</span></pre></div>