"MSWA: Refining Local Attention with Multi-ScaleWindow Attention"

MSWA introduces variable-sized attention windows across transformer heads and layers, enabling efficient capture of both local and long-range dependencies while reducing computational costs in LLMs.

https://arxiv.org/abs/2501.01039

Original Problem 🤔:

→ Standard self-attention in LLMs has quadratic complexity and high memory usage

→ Sliding Window Attention (SWA) uses fixed window sizes, limiting ability to capture varying context lengths

→ Current solutions fail to effectively balance computational efficiency with context modeling

-----

Solution in this Paper 💡:

→ MSWA applies diverse window sizes across heads and layers in the transformer

→ Each layer has attention heads with window sizes varying from w/4 to 2w

→ Shallow layers focus on local context with smaller windows

→ Deeper layers capture longer dependencies with progressively larger windows

→ Implementation requires minimal changes to existing attention mechanisms

-----

Key Insights 🔍:

→ Variable window sizes better match natural language structure

→ Progressive window size increase allows hierarchical context building

→ Memory usage reduced by 15% compared to standard SWA

-----

Results 📊:

→ Reduces perplexity by 1.14 on Wikitext-103 compared to SWA

→ Achieves 0.11 lower bits-per-character on enwik8

→ Maintains 7% lower computational cost than standard attention

→ Shows 7.23% higher accuracy on 5-shot common-sense tasks

Option 1: MSWA makes transformer attention smarter by using different window sizes for different parts of the model.

Option 2: Different attention windows for different layers - MSWA's simple trick for better LLM performance.

Option 3: MSWA lets transformer layers look at text like humans do - focusing on details and big picture simultaneously.

Option 4: Smart window sizing in transformers: how MSWA makes attention more efficient and effective.

Option 5: Think of MSWA as giving each transformer layer its own unique reading glasses.

Option 6: MSWA: When every transformer layer gets its own personal zoom level.

Option 7: It's like giving each part of your LLM a different-sized spotlight - that's MSWA in action.

Option 8: MSWA lets different parts of your model focus on different chunks of text - just like humans do.