AutoIndexer Enables Flexible Token Editing for Causal Language Models via Chains of Edits
Researchers have introduced AutoIndexer, a generative model that allows causal language models to insert, substitute, and delete arbitrary tokens within pre-generated text.
Standard autoregressive language models struggle with non-causal dependencies like cross-references because they cannot revise outputs after generation. While diffusion-based models offer some flexibility, they still lack the capability to dynamically add or remove arbitrary token sequences from existing contexts. To address this, researchers developed AutoIndexer, a model trained on a Chain of Edits mechanism that simulates text editor behavior.
Primary source
Open arXiv ↗This article summarizes the source. Check the original for full details and conditions.
Comparison
| Aspect | Before / Alternative | After / This |
|---|---|---|
| Post-generation Editing | Strict left-to-right generation with no post-hoc revision capabilities | Flexible insertions, substitutions, and deletions using marker tokens and a cursor |
| Context Error Handling | Error propagation through generated context without correction mechanisms | Robust recovery from erroneous contexts via inverse perturbation training |
Action Checklist
- Review the open-source implementation Inspect the repository at https://github.com/AutodeskAILab/autoindexer for training and inference code
- Verify benchmarks and error recovery on targeted tasks Evaluate the model's editing overhead and performance on downstream tasks like code correction before deployment
