Constrained decoding?
Is it possible to implement constraints on the beam during decoding using a seq2seq model? [NeuroLogic Decoding](https://arxiv.org/abs/2010.12884), [Constrained Abstractive Summarization](https://arxiv.org/abs/2010.12723)

I see that there is a Callback feature in the library - but AFAIK it only lets modify the training state and not the beam. 
I think you should take a look at `prefix_allowed_tokens_fn` on the [`generate` method](https://huggingface.co/transformers/main_classes/model.html?highlight=generate#transformers.generation_utils.GenerationMixin.generate). 

It let's you create a function to constrain the generation based on previously generated tokens. It is inspired by this paper:  [Autoregressive Entity Retrieval](https://arxiv.org/abs/2010.00904).<|||||>thanks! Is there a way to have more generic callbacks that can possibly access the beams (for instance, if I want to do custom scoring/tracking/modification to the entries in the beam)?

If not, would that be a useful PR to submit?<|||||>So you mean to access the different beams so that generations is conditioned to not just the current beam so far but also the others? 
You probably should look at transformers.LogitsProcessor in:
https://github.com/huggingface/transformers/blob/1750e629006bb6989aef5b4e141f3477f891a098/src/transformers/generation_utils.py#L555-L629

 which deals with the constraints and scoring of tokens at generation. Perhaps what you described could be introduced in a similar fashion as `prefix_allowed_tokens_fn`.

Regarding a PR I am not the best to say, I would first make sure if what you aim for can be done within the existing functionality. <|||||>I did take a look at `LogitsProcessor` and might be able to work with that - thanks!<|||||>@kailashkarthiks Hi, I am also interested in performing constrained decoding with seq2seq language models like BART. I was wondering if you made any progress on this, and if yes, would you be able to share your findings or code? Thank you!