"MambaByte: Token-free Selective State Space Model"

MambaByte demonstrates token-free language modeling can be competitive with subword approaches while offering improved robustness.

MambaByte is a token-free adaptation of the Mamba SSM trained autoregressively on byte sequences.

📌 In terms of modeling, MambaByte is competitive with SOTA subword Transformers on language modeling tasks while maintaining the benefits of token-free language models, such as robustness to noise.

📌 This results in a 2.6× inference speedup to the standard MambaByte implementation, showing similar decoding efficiency as the subword Mamba.

📚 https://arxiv.org/pdf/2401.13660

Original Problem 💡:

Subword tokenization in language models introduces issues like lack of robustness to typos and morphological variations. Byte-level modeling offers an alternative but results in much longer sequences, creating efficiency challenges for standard autoregressive Transformers.

-----

Solution in this Paper 🔬:

• Proposes MambaByte, adapting the Mamba state space model (SSM) for byte-level language modeling

• Mamba has a fixed-size memory state independent of sequence length, making it suitable for long byte sequences

• Develops speculative decoding with tokenized drafting and byte-level verification to improve inference speed

-----

Key Insights from this Paper 💡:

• MambaByte competitive with or outperforms subword Transformers on language modeling tasks

• More robust to input noise compared to subword models

• Can extrapolate to sequences 4x longer than training length

• Speculative decoding enables generation speed comparable to subword Mamba

-----

Results 📊:

• Outperforms byte-level baselines on PG19 dataset (33.0 vs 36.4 word-level PPL for MegaByte)

• 2.6x faster generation than MegaByte

• Significantly more resilient to synthetic noise (e.g. only +28.3 PPL degradation vs +58300.0 for subword Mamba on "antspeak" noise)