Extending Context Length in Large Language Models

Browse all previously published AI Tutorials here.

Table of Contents

• Extending Context Length in Large Language Models

• Theoretical Advancements in Long-Context Modeling

• Implementation Details and Efficiency Considerations

• Applications and Case Studies with Extended Context

• Comparison of Approaches and Effectiveness

Introduction: Modern LLMs have historically been constrained to modest context windows (e.g. 2K–4K tokens), limiting their ability to handle very long documents in a single pass. Recently, there is intensive research on overcoming the quadratic cost of Transformer attention and the positional limits in training, in order to scale context lengths to tens of thousands of tokens and beyond. This review surveys recent advances (mostly 2023–2025) in model architectures and training techniques that extend LLM context length, along with implementation strategies and real-world use cases for long-context LLMs.

Theoretical Advancements in Long-Context Modeling

Beyond sparsity, recurrence has been reintroduced to Transformers to handle unbounded contexts. Transformer-XL and Compressive Transformer (2019) pioneered segment-level recurrence (carrying hidden states forward) to effectively get unlimited context, albeit with some loss of detail over very long ranges. Recent works build on this: Recurrent Memory Transformer (RMT) and the improved Associative RMT (ARMT) combine local self-attention on each segment with a recurrent memory for long-term dependencies (Associative Recurrent Memory Transformer) . ARMT set a new record on the BABILong benchmark by accurately answering questions that depend on 50 million tokens of context . Such models process chunks in parallel but maintain an “infinite” context through learned memory states, effectively extending beyond fixed windows.

State-Space Models and RNNs: Outside pure Transformers, recurrent architectures are experiencing a revival for long sequence modeling. Recurrent neural networks inherently handle arbitrary sequence lengths by sequential processing. However, classical RNNs struggled with training at scale. New formulations like Structured State-Space Models (SSMs) offer a way to retain RNN-like infinite memory with efficient parallel computation (HERE) . The Mamba architecture (Gu et al., 2024) integrates SSMs into an LLM, removing the attention bottleneck. Mamba achieves linear scaling in sequence length and can utilize contexts up to ~1 million tokens with strong performance (Mamba Explained) . Notably, a 3B-parameter Mamba language model matched or outperformed Transformers twice its size, all while enjoying up to 5× faster generation speeds . This represents a major theoretical breakthrough: Mamba is the first RNN/SSM-based LLM to rival Transformer quality at scale, enabling ultra-long context without quadratic cost. Similarly, Microsoft’s Retentive Network (RetNet) (2023) bridges recurrence and attention by introducing a retention mechanism. RetNet can be trained in parallel like a Transformer, but at inference it runs as an O(1) recurrent model, supporting chunkwise processing of long sequences with linear complexity ([2307.08621] Retentive Network: A Successor to Transformer for Large Language Models) ([2307.08621] Retentive Network: A Successor to Transformer for Large Language Models). Such architectures promise the benefits of both worlds – efficient training and unlimited context length at inference.

Positional Encoding & Extrapolation: Even with efficient attention, LLMs must learn to utilize long contexts. A key limitation is that models are usually pre-trained with fixed-length sequences, so they often fail to generalize beyond that length (HERE). Recent work has focused on positional encoding schemes and training tricks to enable length extrapolation. For instance, relative position embeddings like ALiBi (Attention with Linear Bias) provide a monotonic bias that allows limited generalization beyond the training window . More sophisticated is the use of rotary position embeddings (RoPE) with interpolation. Techniques such as Position Interpolation and NTK-aware scaling adjust RoPE frequency matrices so that a model trained on, say, 2K tokens can be stretched to 8K or 32K at inference with minimal fine-tuning . These were adopted in extending LLaMA-2 and CodeLlama models to 32K contexts (e.g. Code Llama used NTK-aware interpolation) . The YaRN method (Peng et al., 2023) improved on these by fine-tuning on a small amount of longer data and using dynamic positional scaling. YaRN achieved state-of-the-art results in context extension, allowing LLaMA 2 models to extrapolate up to 128K tokens with only 0.1% of original pre-training data and modest fine-tuning . In fact, YaRN-extended LLaMA models showed effective utilization of contexts far beyond their original limit, with LLaMA-2 13B reaching 128K tokens of usable context . Likewise, Shifted RoPE embedding (StRing) (An et al., 2024) addresses the “effective length” gap by dynamically reassigning position indices at inference. Without any retraining, StRing boosted long-context performance of open models by >10 points on benchmarks, even surpassing proprietary models like GPT-4-128K in some tests (Why Does the Effective Context Length of LLMs Fall Short?) .

Mixture-of-Experts (MoE) Approaches: While MoE primarily aims at scaling model capacity, some recent ideas leverage MoE for long-context handling. Mixture of In-Context Experts (MoICE) (Lin et al., 2024) is a novel method that treats different positional representations as experts ([2406.19598] Mixture of In-Context Experts Enhance LLMs' Long Context Awareness). It inserts a learned router in each attention head to dynamically weight multiple RoPE angles, enabling the head to attend to both near and far tokens effectively ([2406.19598] Mixture of In-Context Experts Enhance LLMs' Long Context Awareness) ([2406.19598] Mixture of In-Context Experts Enhance LLMs' Long Context Awareness). By freezing the base model and training only these routers briefly, MoICE significantly improved LLaMA and Mistral models on long-context understanding tasks, without slowing inference ([2406.19598] Mixture of In-Context Experts Enhance LLMs' Long Context Awareness). This highlights a creative use of MoE: instead of splitting neural network parameters, it splits positional attention behavior among experts, thereby enhancing long-range awareness.

Implementation Details and Efficiency Considerations

Extending Training Context: One straightforward approach to longer contexts is to train or fine-tune LLMs on longer sequences. Researchers have used curriculum strategies (gradually increasing sequence lengths during training) and targeted fine-tuning. As noted, methods like YaRN achieved 128K-token training by using only a small fraction of data (HERE). Others (e.g. OpenAI GPT-4, Anthropic Claude) leveraged large-scale training on long sequences once efficient attention mechanisms and sufficient hardware became available. Still, full retraining at long lengths is extremely costly. Therefore, a lot of attention has turned to post-hoc solutions that extend context without retraining. Position interpolation and NTK-scaling are such inference-time tricks. Another example is LM-Infinite (Han et al., 2023) which introduces a simple Λ-shaped attention mask and a distance cap during generation (LM-Infinite: Simple On-the-Fly Length Generalization for Large Language Models | OpenReview). By preventing attention to extremely distant tokens and limiting unseen positional interactions, LM-Infinite enabled off-the-shelf LLMs (with relative positional encodings) to generate coherent text up to 128K tokens long. This method required no parameter updates and ran in O(n) time/space, achieving 2.7× faster decoding while maintaining fluency on long inputs . Such approaches are attractive for deployment since they extend context on pre-trained models in a “plug-and-play” fashion.

Memory and Computational Scaling: Even if an LLM can conceptually handle 100K tokens, the practical GPU memory usage and latency are major concerns. Various optimizations have been proposed. FlashAttention (Dao et al., 2022) and related kernels minimize memory overhead of attention, making long sequences more feasible by tiling computations ([2502.08910v1.pdf](file://file-PFghuvAEVtWpQpXoLVPTFQ#:~:text=Dao%2C%20T,awareness%2C%202022.%20URL%20http%3A%2F%2Farxiv.%20org%2Fabs%2F2205.14135)). On the inference side, Flash-Decoding (2023) reorders attention computation to reduce latency for long contexts . A different approach is to offload or compress the context. For example, the key-value cache (KV) of the attention mechanism grows with sequence length; offloading it to CPU or disk can trade speed for capacity. The InfiniteHoP (Infinite Hierarchical Pruning) framework (Lee et al., 2025) combines several such ideas. It prunes away tokens deemed irrelevant using a hierarchical algorithm, and selectively applies RoPE adjustments on the fly to maintain performance ([2502.08910] InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU) ([2502.08910] InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU). It also offloads the KV cache to host memory. Together, these enable processing up to 3 million tokens on a single 48GB GPU with no loss of context information ([2502.08910] InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU) ([2502.08910] InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU). Notably, InfiniteHiP achieved an 18.9× speedup in attention decoding for a 1M-token input, all without any model retraining ([2502.08910] InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU). This kind of system-level solution is critical for making ultra-long contexts usable in practice, by dynamically shrinking or externalizing parts of the context.

In summary, implementation research has focused on keeping the effective context large while controlling compute costs. Techniques range from sparse attention patterns (to skip computation on far-away pairs), chunkwise processing (to reuse summary states and avoid quadratic blowup), cache management (offloading or quantizing memory), and smart positional hacks (rescaling or shifting positions to exploit model biases). As a result, today’s models can be deployed with context windows an order of magnitude larger than those seen during training, albeit with careful engineering.

Applications and Case Studies with Extended Context

Ultra-long context LLMs unlock new use cases in document processing, code analysis, and beyond. A prime example is document digitization and analysis. Instead of breaking a long document into chunks for separate processing (with external tools to stitch together answers), an LLM with a 100K+ token window can ingest hundreds of pages at once. Anthropic demonstrated this by having Claude read the entire Great Gatsby (~72K tokens) in one go and correctly identify a subtle altered sentence in seconds (Introducing 100K Context Windows \ Anthropic) . This capability is transforming how we approach tasks like summarization, where previously a book or lengthy report would need to be processed chapter by chapter. Now, models like Claude 2 (100K context) or specialized long-context LLaMA variants can directly answer questions that require synthesis across an entire book or multi-document collection . In a business setting, this means an LLM assistant could ingest an entire corporate annual report or a lengthy contract and answer detailed queries without missing context hidden in earlier pages.

Another compelling case study is in audio and meeting transcription analysis. With extended context, one can feed an LLM a very long transcription (say, a day’s worth of meeting logs or a multi-hour podcast) and obtain summaries or perform Q&A in one shot. Anthropic’s partners at AssemblyAI showcased this by transcribing a long podcast (~58K words) and using Claude to summarize it and answer questions – tasks that fit comfortably in Claude’s 100K token window . The result is a simple pipeline: speech-to-text followed by a single LLM query for summarization, eliminating the need for splitting the transcript or doing heuristic keyword retrieval. Similarly, developers have experimented with ingesting entire codebases (tens of thousands of lines of code) into an LLM context for code understanding and generation tasks . For example, a long-context model can take all files of a software project and then implement a new feature that touches multiple modules, referring to the global context as needed – something that standard 4K-token models would struggle with unless code was provided piecemeal.

Document chunking vs. long-context processing: In traditional workflows, chunking large inputs was necessary: a document would be split into overlapping chunks, an LLM would process each, and an external logic (like retrieval or iterative summarization) would combine the results. Extended context models promise to simplify this. Early evidence suggests that for certain complex queries, a single long context may outperform retrieval-based approaches because the model can reason over all relevant pieces simultaneously (Introducing 100K Context Windows \ Anthropic). However, there are trade-offs – extremely long inputs can incur high computation cost, and not all models truly utilize the extra context efficiently. In practice, users often adopt a hybrid approach: use an LLM with as large a context as possible, and only resort to chunking if the input exceeds that limit or if using a smaller model for cost reasons. The development of benchmarks like LongBench (2023, avg 10K tokens) and ∞Bench (2024, 100K+ tokens) is helping quantify these advantages (∞Bench: Extending Long Context Evaluation Beyond 100K Tokens) . Evaluations on ∞Bench showed that even state-of-the-art long-context models still struggle to fully retain and reason over 100K+ tokens of information . This underlines that simply having a large context window is not enough – the model architecture and training must foster long-range attention and memory.

Comparison of Approaches and Effectiveness

Different methods for extending context come with distinct strengths and limitations:

• Transformer-based vs. Recurrent: Pure Transformer solutions (like LongNet’s dilated attention) maintain parallelism and are straightforward to train, but some capacity to model extremely long dependencies might be sacrificed (e.g. LongNet relies on fixed dilation patterns). Recurrent or hybrid models (RetNet, RMT/ARMT, Mamba) can, in principle, handle infinite sequences and have fast autoregressive inference ([2307.08621] Retentive Network: A Successor to Transformer for Large Language Models) (Mamba Explained). They often achieve linear time complexity per token, making them attractive for deployment. However, they introduce complexity in training (chunkwise or memory-augmented training regimes) and are relatively new – further scaling and fine-tuning will reveal if they can fully replace Transformers.

• Architectures for Long Range: State-space models (Mamba) and gated RNNs (RetNet) have demonstrated no loss in performance even at 1M token contexts , a promising result. Meanwhile, extended Transformers with position interpolation (YaRN, NTK methods) have shown that existing LLMs can be pushed to 32K–128K with minimal fine-tuning (HERE), but beyond that, quality may degrade without architectural changes. Mixture-of-Experts like MoICE offer a lightweight fine-tuning to boost long-context awareness (reducing the tendency to ignore distant tokens) ([2406.19598] Mixture of In-Context Experts Enhance LLMs' Long Context Awareness), complementing other methods.

• Efficiency and Practicality: Methods that require training from scratch for long contexts (e.g. LongNet trained to 1B tokens, or a new 100K-token Transformer) are expensive but yield models inherently designed for those lengths. In contrast, inference-time tricks (LM-Infinite’s masks (LM-Infinite: Simple On-the-Fly Length Generalization for Large Language Models | OpenReview), StRing shifts (Why Does the Effective Context Length of LLMs Fall Short?), InfiniteHiP pruning ([2502.08910] InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU)) are highly practical for adapting existing models and can be toggled on as needed. The trade-off often comes in inference speed and memory: for example, InfiniteHiP can reach millions of tokens by trading GPU memory for CPU offloading and accepting some increase in latency ([2502.08910] InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU). Users must balance context length requirements with available resources. It’s also observed that many open-source long-context models have an “effective context” much shorter than the theoretical maximum if not carefully trained . Thus, the most effective strategy might combine approaches – e.g. use a model trained on moderately long contexts and apply an inference method to safely extrapolate further.

In conclusion, the landscape of extending LLM context length is evolving rapidly. Transformer-based innovations (dilated or sparse attention, better position encodings) have significantly pushed the limits, with models now boasting 100K or even billion-token windows. Recurrent and state-space architectures offer a compelling alternative by fundamentally sidestepping the attention bottleneck and treating long sequences as a first-class citizen. Mixture-of-experts and memory techniques add new dimensions by allocating model capacity to different context segments or external memory. Real-world deployments already illustrate the value of long-context LLMs in handling tasks like document comprehension, code analysis, and multi-document QA that were previously cumbersome with short-context models. As research continues, we can expect these methods to be refined and combined, moving closer to the goal of truly infinite context language models (Mamba Explained - The Gradient) – models that can read and reason over entire libraries of text seamlessly. The progress from 2023 to 2025 suggests that this ambitious vision is on the horizon, backed by both theoretical insights and engineering breakthroughs in large-scale sequence modeling.