<h1 dir="auto">🚀 Feature request</h1>
<p dir="auto">The current <a href="https://github.com/huggingface/transformers/blob/3977b58437b8ce1ea1da6e31747d888efec2419b/src/transformers/pipelines/token_classification.py#L82">implementation of TokenClassificationPipeline</a> cannot do anything with text input beyond the length (measured in tokens) allowed by the model's number of positional embeddings. I think it would be useful to many users to have a setting in the pipeline that allows for a "sliding window" approach that can take documents longer than the <code class="notranslate">.config.max_position_embeddings</code> of the model in use.</p>
<p dir="auto">The modified version of <code class="notranslate">TokenClassificationPipeline</code> could be instantiated with the optional new params <code class="notranslate">window_size</code> (defaults to <code class="notranslate">model.max_position_embeddings - 2</code> to accomodate the <code class="notranslate">[CLS]</code> and <code class="notranslate">[SEP]</code> special tokens) and <code class="notranslate">stride</code> (defaults to <code class="notranslate">window_size / 2</code>, the user can disable the sliding window by setting <code class="notranslate">stride = 0</code>) like so:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="model, tokenizer = # Some model, some tokenizer
pipeline = TokenClassificationPipeline(model=model, tokenizer=tokenizer, aggregation_strategy='first', stride=254)"><pre class="notranslate"><code class="notranslate">model, tokenizer = # Some model, some tokenizer
pipeline = TokenClassificationPipeline(model=model, tokenizer=tokenizer, aggregation_strategy='first', stride=254)
</code></pre></div>
<h2 dir="auto">Motivation</h2>
<p dir="auto">I run tasks that involve doing named entity recognition on longer documents. I have to use my own modified subclass of <code class="notranslate">TokenClassificationPipeline</code> in order to do so, since the existing class does not support processing texts longer than the maximum number of position embeddings for the underlying models.</p>
<h2 dir="auto">Your contribution</h2>
<p dir="auto">I will submit a PR adding sliding window functionality to <code class="notranslate">TokenClassificationPipeline</code>. I have previously created a sliding window implementation of Transformers' NER Pipeline <a href="https://github.com/nlpsandbox/phi-annotator-huggingface/blob/634e1736837b50a190f42c2fc741b262e88bf3ac/server/openapi_server/huggingface.py#L15">here</a>, and will re-work this to work with TensorFlow and meet the code standards of HuggingFace/Transformers.</p>
<p dir="auto">Hi, I am trying to convert a fine-tuned GPT-Neo (125M) model to ONNX using the code below:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from transformers import pipeline, convert_graph_to_onnx, GPTNeoForCausalLM, GPT2Tokenizer
from pathlib import Path
import torch

model_name = &quot;EleutherAI/gpt-neo-125M&quot; 
pipeline_name = &quot;text-generation&quot;

tokenizer = GPT2Tokenizer.from_pretrained(&quot;EleutherAI/gpt-neo-125M&quot;, bos_token='&lt;|startoftext|&gt;',
                                                          eos_token='&lt;|endoftext|&gt;', pad_token='&lt;|pad|&gt;')
nlp = pipeline(pipeline_name, model=model_name, tokenizer=tokenizer)

with torch.no_grad():
    (
        input_names,
        output_names,
        dynamic_axes,
        tokens,
    ) = convert_graph_to_onnx.infer_shapes(nlp, &quot;pt&quot;)
    ordered_input_names, model_args = convert_graph_to_onnx.ensure_valid_input(
        nlp.model, tokens, input_names
    )

model_name = 'gpt-neo'

predictor_path = './' + model_name 

model2 = GPTNeoForCausalLM.from_pretrained(predictor_path)

text = &quot;I feel happy and &quot;
prompt = f'&lt;|startoftext|&gt;Review: {text} Sentiment: &lt;|endoftext|&gt;'
encodings_dict = nlp.tokenizer(prompt, truncation=True, max_length=300, padding=&quot;max_length&quot;, return_tensors=&quot;pt&quot;)

torch.onnx.export(
    model2,
    (encodings_dict['input_ids'], encodings_dict['attention_mask']),
    'model_test.onnx',
    input_names=input_names,
    output_names=output_names,
    dynamic_axes=dynamic_axes,
    do_constant_folding=True,
    use_external_data_format=True, # Needed because of model size
    enable_onnx_checker=True,
    opset_version=13
)
"><pre class="notranslate"><code class="notranslate">from transformers import pipeline, convert_graph_to_onnx, GPTNeoForCausalLM, GPT2Tokenizer
from pathlib import Path
import torch

model_name = "EleutherAI/gpt-neo-125M" 
pipeline_name = "text-generation"

tokenizer = GPT2Tokenizer.from_pretrained("EleutherAI/gpt-neo-125M", bos_token='&lt;|startoftext|&gt;',
                                                          eos_token='&lt;|endoftext|&gt;', pad_token='&lt;|pad|&gt;')
nlp = pipeline(pipeline_name, model=model_name, tokenizer=tokenizer)

with torch.no_grad():
    (
        input_names,
        output_names,
        dynamic_axes,
        tokens,
    ) = convert_graph_to_onnx.infer_shapes(nlp, "pt")
    ordered_input_names, model_args = convert_graph_to_onnx.ensure_valid_input(
        nlp.model, tokens, input_names
    )

model_name = 'gpt-neo'

predictor_path = './' + model_name 

model2 = GPTNeoForCausalLM.from_pretrained(predictor_path)

text = "I feel happy and "
prompt = f'&lt;|startoftext|&gt;Review: {text} Sentiment: &lt;|endoftext|&gt;'
encodings_dict = nlp.tokenizer(prompt, truncation=True, max_length=300, padding="max_length", return_tensors="pt")

torch.onnx.export(
    model2,
    (encodings_dict['input_ids'], encodings_dict['attention_mask']),
    'model_test.onnx',
    input_names=input_names,
    output_names=output_names,
    dynamic_axes=dynamic_axes,
    do_constant_folding=True,
    use_external_data_format=True, # Needed because of model size
    enable_onnx_checker=True,
    opset_version=13
)

</code></pre></div>
<p dir="auto">But I get this error:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="ValueError                                Traceback (most recent call last)
&lt;ipython-input-15-024e093371a4&gt; in &lt;module&gt;()
     43     use_external_data_format=True, # Needed because of model size
     44     enable_onnx_checker=True,
---&gt; 45     opset_version=13
     46 )

3 frames
/usr/local/lib/python3.7/dist-packages/torch/onnx/utils.py in _validate_dynamic_axes(dynamic_axes, model, input_names, output_names)
   1300             for i, x in enumerate(value):
   1301                 if not isinstance(x, int):
-&gt; 1302                     raise ValueError(&quot;The type of axis index is expected to be an integer&quot;)
   1303                 if x in value_dict:
   1304                     warnings.warn(&quot;Duplicate dynamic axis index {} was provided for input {}.&quot;

ValueError: The type of axis index is expected to be an integer"><pre class="notranslate"><code class="notranslate">ValueError                                Traceback (most recent call last)
&lt;ipython-input-15-024e093371a4&gt; in &lt;module&gt;()
     43     use_external_data_format=True, # Needed because of model size
     44     enable_onnx_checker=True,
---&gt; 45     opset_version=13
     46 )

3 frames
/usr/local/lib/python3.7/dist-packages/torch/onnx/utils.py in _validate_dynamic_axes(dynamic_axes, model, input_names, output_names)
   1300             for i, x in enumerate(value):
   1301                 if not isinstance(x, int):
-&gt; 1302                     raise ValueError("The type of axis index is expected to be an integer")
   1303                 if x in value_dict:
   1304                     warnings.warn("Duplicate dynamic axis index {} was provided for input {}."

ValueError: The type of axis index is expected to be an integer
</code></pre></div>
<p dir="auto">But if I remove dynamic_axes, I get this:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="IndexError                                Traceback (most recent call last)
&lt;ipython-input-16-89ddee8c10f8&gt; in &lt;module&gt;()
     43     use_external_data_format=True, # Needed because of model size
     44     enable_onnx_checker=True,
---&gt; 45     opset_version=13
     46 )

15 frames
/usr/local/lib/python3.7/dist-packages/transformers/models/gpt_neo/modeling_gpt_neo.py in forward(self, input_ids, past_key_values, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, use_cache, output_attentions, output_hidden_states, return_dict)
    545             past_key_values = tuple([None] * len(self.h))
    546         else:
--&gt; 547             past_length = past_key_values[0][0].size(-2)
    548 
    549         device = input_ids.device if input_ids is not None else inputs_embeds.device

IndexError: dimension specified as -2 but tensor has no dimensions"><pre class="notranslate"><code class="notranslate">IndexError                                Traceback (most recent call last)
&lt;ipython-input-16-89ddee8c10f8&gt; in &lt;module&gt;()
     43     use_external_data_format=True, # Needed because of model size
     44     enable_onnx_checker=True,
---&gt; 45     opset_version=13
     46 )

15 frames
/usr/local/lib/python3.7/dist-packages/transformers/models/gpt_neo/modeling_gpt_neo.py in forward(self, input_ids, past_key_values, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, use_cache, output_attentions, output_hidden_states, return_dict)
    545             past_key_values = tuple([None] * len(self.h))
    546         else:
--&gt; 547             past_length = past_key_values[0][0].size(-2)
    548 
    549         device = input_ids.device if input_ids is not None else inputs_embeds.device

IndexError: dimension specified as -2 but tensor has no dimensions
</code></pre></div>
<p dir="auto">Can someone help me please?</p>