For many models, like GPT2, the generate function accepts bad_words_ids. We’re currently passing about 2500 tokenized phrases into this, and finding that it works well, but also finding that it slows down inference considerably… with 2500 phrases, we find that a generate that would take 250ms without the bad_words_ids, takes about 500ms.
Maybe there is just no solution for this, and we need to simply curtail our usage of bad_words_ids?
We were also looking at this code:

  

      github.com
  

  
    huggingface/transformers/blob/b1198a8440cc05f569b0bc22038993a1e5e707ab/src/transformers/generation_logits_process.py#L345 1



      
      
                      )
                      for hypo_idx in range(num_hypos)
                  ]
          
          
        for i, banned_tokens in enumerate(banned_batch_tokens):
                      scores[i, banned_tokens] = -float("inf")
          
          
        return scores
          
          

          
class NoBadWordsLogitsProcessor(LogitsProcessor):
              """
              :class:`transformers.LogitsProcessor` that enforces that specified sequences will never be sampled.
          
          
    Args:
                  bad_words_ids (:obj:`List[List[int]]`):
                      List of list of token ids that are not allowed to be generated. In order to get the tokens of the words
                      that should not appear in the generated text, use :obj:`tokenizer(bad_word,
                      add_prefix_space=True).input_ids`.
                  eos_token_id (:obj:`int`):
                      The id of the `end-of-sequence` token.
      
    
  


  

  
    
    
  

  


A the bad_words_ids are required to be passed in as a list. If somehow we could use a tensor instead, and put that tensor on the GPU, would that speed it up?
Any other suggestions?
thanks