"Gated Delta Networks: Improving Mamba2 with Delta Rule"

Memory management in transformers just got an upgrade with gated delta networks

Gated DeltaNet combines gating mechanisms with delta rule updates to enhance memory management in linear transformers, enabling better performance across language modeling tasks.

-----

https://arxiv.org/abs/2412.06464

🤔 Original Problem:

Linear Transformers face challenges in managing information over long sequences and retrieval tasks. They struggle with memory collisions when sequence length exceeds model dimensionality, while existing solutions like Mamba2 and DeltaNet have limitations in memory management.

-----

🔧 Solution in this Paper:

→ The paper introduces the gated delta rule that combines both gating and delta updates for flexible memory control

→ It implements a parallel training algorithm using WY representation to maintain computational efficiency

→ The architecture allows rapid memory clearing through gating (α_t → 0) while enabling selective updates through delta rule (α_t → 1)

→ Gated DeltaNet further enhances performance by developing hybrid versions that combine with sliding window attention or Mamba2 layers

-----

💡 Key Insights:

→ Gating enables rapid memory erasure while delta rule facilitates targeted updates

→ The hybrid approach improves both training efficiency and task performance

→ The gated delta rule provides more flexible memory control than either mechanism alone

-----

📊 Results:

→ Consistently outperforms Mamba2 and DeltaNet across benchmarks

→ Achieves 21.19% annual return rate with 0.62 Sharpe ratio using MOM 7M factor

→ Maintains same training speed as DeltaNet with only 2-3K tokens/sec slower than Mamba2

→ Hybrid versions show superior performance in long-context understanding tasks