Constrained Decoding: Forcing LLMs to Respect Your Taxonomy

The correctness proof for trie-based constrained decoding hinges on maintaining two key invariants throughout the entire generation process:

1. Forward Invariant: Validity of Tokens at Each Step

At each decoding step (t), the processor ensures that the set of valid tokens (those not masked to (-\infty)) consists only of tokens that would continue a valid path through the trie. This invariant is maintained by:

• Trie Querying: The processor queries the trie for valid next tokens based on the current sequence of generated tokens.
• Masking Invalid Tokens: All tokens that do not form part of any valid continuation are masked to (-\infty), ensuring they cannot be sampled.

2. Backward Invariant: Completeness and Termination

The backward invariant ensures that if a complete label has been formed, the model can either terminate by emitting an EOS token or continue with another label via a separator token:

• EOS Token: If the current sequence of tokens forms a complete label (as determined by the trie), the processor adds the EOS token to the set of valid tokens.
• Separator Handling: When continuing to generate additional labels, already emitted labels are excluded from future selections at the

Read the full article at Towards AI - Medium

---

Want to create content about this topic? Use Nemati AI tools to generate articles, social posts, and more.