Sequential constraints?
fairseq has an implementation of unordered and ordered multi-token constraints: see their [PR](https://github.com/pytorch/fairseq/pull/2402) and [example](https://github.com/pytorch/fairseq/tree/1bba712622b8ae4efb3eb793a8a40da386fe11d0/examples/constrained_decoding).

This is more advanced than the single-token constraints that have been occasionally [requested here](https://github.com/huggingface/transformers/issues/10485) mainly due to the bookkeeping involved; see the papers referenced in the fairseq PR.

Has anyone looked into porting this feature? fairseq's [constraint tracking logic](https://github.com/pytorch/fairseq/blob/master/fairseq/token_generation_constraints.py) looks to be well-factored and could probably be adopted verbatim, license permitting. The beam search modifications ([fairseq implementation](https://github.com/pytorch/fairseq/blob/ee0d5a0f65a25e5f5372776402aac5cb9c4adbf1/fairseq/search.py#L210)) may be able to be implemented as a `LogitsProcessor`, or maybe even just a `prefix_allowed_tokens_fn`, but the papers propose some additional logic around making sure that a partial constraint stays in the beam; I'm not sure whether those hooks are sufficient to implement that logic (or how essential it is to the functionality).

(I've found one [related issue](https://github.com/huggingface/transformers/issues/1163).)
Has anyone working on this yet?<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>If no one else beats me to it, I may tackle this mid-summer. No promises. <|||||>@kcarnold any update? 

There is another recent paper on constrained decoding with complex constraints (fairseq only has positive constraints): [NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraints
](https://arxiv.org/abs/2010.12884)