"Hansel: Output Length Controlling Framework for Large Language Models"

Special tokens act as word-count checkpoints, helping LLMs track and control output length precisely.

The Hansel framework enables precise control over LLM output length through special tokens that track remaining words, without modifying model architecture or compromising generation quality.

-----

https://arxiv.org/abs/2412.14033

🤔 Original Problem:

→ LLMs struggle with precise output length control, impacting real-world applications like news summaries and voice assistants where specific lengths are crucial

→ Existing solutions require architectural changes during pre-training, making them impractical for large models

-----

💡 Solution in this Paper:

→ Hansel introduces periodic special tokens during finetuning that indicate remaining word count

→ Special token |x⟩⟨y| shows Δx+y words remaining until target length

→ Tokens inserted every Δ words with residual parameter δ allowing natural completion

→ Framework compatible with various positional encoding methods including Rotary, ALiBi, and T5 bias

-----

🔍 Key Insights:

→ LLMs inherently struggle with counting, making explicit word tracking necessary

→ Simple special tokens can effectively guide output length without architectural changes

→ Residual parameter prevents abrupt sentence termination

→ Framework generalizes well to unseen target lengths

-----

📊 Results:

→ 75% reduction in Mean Absolute Error compared to prompt-based methods

→ Maintains ROUGE-L and G-Eval scores, showing no quality degradation

→ Zero instances of infinite generation compared to 8.4/10,000 in baseline

→ Successful length control with just 50 instruction samples for new tasks