Batch Inference at Scale: Processing Millions of Text Inputs Efficiently

Browse all previoiusly published AI Tutorials here.

Table of Contents

• Introduction

• High-Throughput Batch Inference Pipelines

  • Micro-Batching and Dynamic Batching

  • Concurrency and Throughput

• GPU Memory Utilization and Batch Size Tuning

  • Batch Size Trade-offs

  • Queuing and Asynchronous Scheduling

• Frameworks and Tools for Batch Inference

  • vLLM

  • Ray Serve

  • NVIDIA Triton Inference Server

  • TorchServe

• Framework Comparison

• Conclusion

Introduction

Processing millions of text inputs at scale demands highly optimized inference pipelines. Modern natural language models (from classification transformers to large language models) achieve highest throughput when fed with batches of inputs, leveraging parallelism on GPUs. The challenge is to maximize throughput and hardware utilization without introducing prohibitive latency or cost. This post dives into the latest (2024–2025) techniques for high-throughput batch inference – covering micro-batching, concurrency tuning, and memory-aware scheduling – with concrete references to production-grade frameworks like vLLM, Ray Serve, NVIDIA Triton, and TorchServe. We explore how these systems implement batching strategies, asynchronous pipelines, and GPU concurrency to efficiently serve use cases such as marketing analytics, social media monitoring, or log analysis at massive scale. All discussion is grounded in current implementation details and real deployment practices, focusing purely on high-density technical insights.

High-Throughput Batch Inference Pipelines

Micro-Batching and Dynamic Batching

Hardware accelerators (GPUs, NPUs) are optimized for vectorized operations on batches of data. In inference serving, dynamic batching (a.k.a. micro-batching) is a core technique: incoming requests are buffered briefly and combined into batch tensors for a single forward pass (Dynamic Request Batching — Ray 2.44.1). By processing, say, 32 text inputs together instead of one at a time, the model amortizes memory access and compute overheads across inputs, dramatically increasing throughput per second. Ray Serve’s documentation succinctly describes this: when a request arrives, the server enqueues it and waits for additional requests to form a batch, then executes one batched inference and splits the results back to each caller . NVIDIA Triton Server implements similar dynamic batcher logic at the C++ level – it can automatically merge individual inference requests into a larger batch to maximize GPU utilization (Scaling ML Workloads Using Nvidia Triton Inference Server | QBurst Blog). This dynamic batching can be configured with a small timeout window (e.g. a few milliseconds) so that the system doesn’t wait too long; Triton allows setting a max queue delay in microseconds to control how long to wait for additional requests , and Ray Serve provides a batch_wait_timeout_s parameter for the same purpose . In practice, these micro-batching techniques significantly boost throughput without large latency penalties. For example, continuous batching (an advanced form of dynamic batching discussed later) can yield over an order-of-magnitude throughput improvement in LLM serving while even reducing median latency under high load (Achieve 23x LLM Inference Throughput & Reduce p50 Latency) .

Micro-batching is effective for both real-time and offline inference scenarios. In real-time serving (many clients sending single requests), frameworks use online dynamic batching – aggregating live requests in-memory – to maximize throughput per model instance (Optimize your inference jobs using dynamic batch inference with TorchServe on Amazon SageMaker | AWS Machine Learning Blog) . In offline or batch processing jobs (millions of texts processed from storage), the strategy might simply be to shard the data and feed large pre-made batches to the model. In both cases, the underlying principle is the same: use the largest batch sizes that hardware and latency constraints allow. Batching saturates the parallel processing capacity of GPUs, often delivering higher throughput per dollar by keeping expensive accelerators busy . Modern inference servers have native support for this. TorchServe, for instance, was designed to natively batch incoming requests – the server will aggregate requests up to a configured batch_size and dispatch them together, yielding optimal use of the GPU and reducing overall cost per inference (Batch Inference with TorchServe — PyTorch/Serve master documentation) . The key is that the batching is dynamic: the system decides in real-time how to group requests (up to the max batch size or timeout) rather than requiring the client to send pre-batched inputs.

Concurrency and Throughput

Batching alone is not enough; high-throughput systems also exploit concurrency in multiple forms. One form is concurrent model execution – running multiple inference jobs in parallel when hardware resources permit it. NVIDIA Triton supports spinning up multiple instances of the same model (e.g. two copies of a model on one GPU or on different GPUs) to handle queries in parallel (Dynamic Batching & Concurrent Model Execution — NVIDIA Triton Inference Server). This is configured via instance groups in Triton’s model config, and it’s useful if a single model copy can’t fully utilize the GPU or to serve more requests concurrently. By load-balancing requests over two model instances, Triton can nearly double throughput in some cases (assuming the GPU had spare compute headroom) and reduce queue wait times for each request . Essentially, multiple model workers process batches concurrently, improving pipeline throughput. TorchServe similarly allows configuring multiple workers per model, each a separate process that can handle its own batch of requests – increasing throughput at the cost of extra memory (each worker loads the model) . In distributed serving setups, Ray Serve can scale out replicas of a deployment across many nodes, achieving concurrency via parallel actors each handling batched requests. In all cases, the idea is to use more parallelism when a single thread or single model instance isn’t enough to handle the request volume.

Another form of concurrency is pipeline parallelism and asynchronous I/O. High-throughput text inference pipelines often overlap different stages of processing. For example, while the GPU is busy computing a batch, the CPU can simultaneously be reading the next batch of text inputs and tokenizing them. Frameworks that utilize async event loops (like Ray Serve’s asyncio-based batch handling (Dynamic Request Batching — Ray 2.44.1)) can naturally overlap waiting for new requests with ongoing inference. This reduces idle time. The goal is to keep every component of the system – from input queues, to GPUs, to network – as busy as possible doing useful work. Tuning concurrency involves finding the right number of parallel requests to keep in flight. Too little concurrency (e.g. one request at a time) underutilizes resources, while too much can lead to long queues and memory contention. A common strategy is to push throughput to the maximum and only back off if latency or memory usage violates requirements. Concretely, in batch inference jobs you would increase the batch size or number of parallel workers until adding more no longer increases throughput (i.e. the GPU is fully utilized) (Optimize your inference jobs using dynamic batch inference with TorchServe on Amazon SageMaker | AWS Machine Learning Blog). For online serving, you adjust batch timeout or number of replicas until the latency is just under your SLA. Modern serving systems often have autoscaling and backpressure mechanisms to manage this concurrency dynamically.

GPU Memory Utilization and Batch Size Tuning

Batch Size Trade-offs

Choosing the optimal batch size is a critical tuning decision for batch inference. Larger batches generally improve throughput (images or text tokens per second) due to better hardware utilization, but they also increase per-request latency and memory usage (Do Large Batches Always Improve Neural Network Throughput?). There is an inflection point beyond which a batch that is too large will saturate GPU memory or start to add overhead (diminishing returns on throughput). Recent research on LLM serving emphasizes that throughput, latency, and memory are interdependent with respect to batch size (Optimizing LLM Inference Throughput via Memory-aware and SLA ...). In practice, engineers find a sweet spot: for example, using batch sizes of 8 or 16 might double throughput vs. single requests with only a slight latency increase, whereas jumping to 64 could exhaust GPU memory or inflate latency beyond acceptable levels (LLM Inference Performance Engineering: Best Practices - Databricks). Monitoring GPU utilization and latency percentiles is essential when tuning batch size. If the GPU isn’t near 100% utilization during inference, you can likely increase batch size to use more parallel capacity. If latency or memory usage is the concern, you might use a smaller batch or even slice a large batch into micro-batches internally to stream through the model.

Memory optimization can effectively increase the feasible batch size. Large NLP models, especially LLMs, are memory-bound (storing attention keys/values for each input token). Techniques like PagedAttention in vLLM compress the memory footprint per sequence by using a paging allocator for the KV cache (Achieve 23x LLM Inference Throughput & Reduce p50 Latency). This means you can fit more text sequences into GPU memory at once without running out of memory, thereby enabling a higher batch size and higher throughput . In fact, vLLM’s authors noted that their memory-efficient approach allows serving significantly more concurrent text generation requests, directly translating into throughput gains . Another example is using lower precision (FP16 or INT8 quantization) which cuts memory per input and speeds up each forward pass, often enabling larger effective batches. The trade-off is that any increase in batch size will add some latency for individual requests – but for batch processing of millions of texts, latency per item is usually a secondary concern to total throughput. Ultimately, careful batch size tuning and memory management can minimize cost per input processed by fully utilizing the GPU’s capability with as many parallel texts as possible in each inference cycle.

Queuing and Asynchronous Scheduling

High-throughput inference servers rely on smart queuing strategies to balance latency and throughput. As discussed, most frameworks implement a configurable waiting period to accumulate batchable requests. If traffic is low, the server will wait only a brief moment (for example, 50 ms in TorchServe’s default config (Batch Inference with TorchServe — PyTorch/Serve master documentation) or a few hundred microseconds in Triton (Dynamic Batching & Concurrent Model Execution — NVIDIA Triton Inference Server)) to see if more requests arrive to fill the batch; if the batch is still not full when the timer expires, it proceeds with whatever requests it has. This ensures that occasional requests don’t get stuck waiting too long, while at high load the server naturally batches many together. The queue behavior must be carefully tuned: a longer max wait increases batch size (better throughput) but also adds latency for early arrivals, whereas a zero wait (always immediately infer) would minimize latency but forgo most batching benefits. For example, TorchServe allows setting maxBatchDelay in milliseconds to control this wait . Ray Serve’s batch_wait_timeout_s similarly lets you adjust how long to gather a micro-batch before timing out (Dynamic Request Batching — Ray 2.44.1). In production, these values are often set very small (tens of milliseconds or less) since even a short wait is enough to batch many concurrent requests under heavy load.

Asynchronous scheduling is another cornerstone of throughput-oriented design. Nearly all modern inference servers use async I/O or multi-threaded scheduling to pipeline tasks. Ray Serve’s internal asyncio event loop batches requests in an async function so that the thread can handle other work while waiting . NVIDIA Triton employs a scheduler thread per model that continuously pulls requests from a queue, forms batches, and dispatches them to GPU execution. The scheduling is non-blocking – multiple batches across different models or model instances can be formed and executed in parallel if resources allow . This asynchronous design also enables overlapping of compute and communication: e.g., while one batch is running on the GPU, the scheduler can start pre-processing the next batch and enqueue it, achieving near-zero idle time on the accelerator. Some specialized systems even implement iteration-level scheduling for auto-regressive models: interleaving the decode steps of multiple text generation requests. This is what vLLM and Hugging Face’s Text Generation Inference (TGI) server do – they continuously add new requests into an ongoing generation loop, rather than waiting for the current batch to finish completely (LLM Inference at scale with TGI). Such continuous batching ensures the GPU never sits idle between decoding iterations, which is crucial for LLMs that generate tokens sequentially. Overall, through careful queuing and async scheduling, inference pipelines keep throughput high: requests are efficiently packed into each GPU run, and any spare moment is used to prepare or execute other tasks.

Frameworks and Tools for Batch Inference

vLLM

vLLM is a high-throughput inference engine tailored for large language models. Released in 2023 by UC Berkeley, it introduced an advanced continuous batching design for text generation. Unlike traditional servers that batch only at request start, vLLM performs iteration-level batching, meaning it can merge different requests at each decoding step of the LLM (LLM Inference at scale with TGI). This is enabled by a custom scheduling algorithm and memory management system. vLLM uses a technique called PagedAttention to manage the GPU memory for the attention cache efficiently (Achieve 23x LLM Inference Throughput & Reduce p50 Latency). Instead of pre-allocating a giant contiguous cache for each sequence (most of which might be empty if sequences terminate early), vLLM allocates memory in pages on demand. This reduces fragmentation and frees up space, allowing far more sequences to be served concurrently . The result is that vLLM can pack the GPU with a larger batch of text prompts than otherwise possible, dramatically improving throughput. In benchmarks, vLLM achieved over 2x the throughput of naive batching on the same model by virtue of these optimizations . The trade-off is a bit more complex scheduling, but vLLM is optimized to keep latency low even with continuous batching. It essentially maximizes token throughput – one study showed vLLM sustaining ~1900 tokens/sec on a single GPU until saturation, far outpacing other serving systems under the same conditions . vLLM offers a serve backend that can be used standalone, and it also integrates with Ray Serve for distributed scaling (Ray can launch vLLM workers on multiple nodes) . In summary, vLLM is a specialized solution focused on batch serving of LLMs, trading a small increase in per-token latency for massive gains in total throughput by smart batching and memory reuse.

Ray Serve

Ray Serve is a scalable model serving framework that emphasizes Pythonic flexibility and distributed execution on a Ray cluster. It is not limited to NLP, but it has strong support for batch inference patterns. Ray Serve’s key feature for throughput is its dynamic request batching decorator, which, as described earlier, allows a deployment to automatically batch incoming calls up to a max_batch_size with a configurable wait time (Dynamic Request Batching — Ray 2.44.1). This means you can write a handler function that accepts a list of text inputs and have Ray Serve handle the queuing and batching behind the scenes. One of Ray Serve’s advantages is that you can embed custom preprocessing, business logic, or postprocessing in the Python handler and still benefit from batching, whereas with lower-level servers like Triton you typically need to conform to model inputs/outputs only. Ray Serve uses an async event loop per replica to gather and execute batches efficiently . It also supports concurrency tuning: you can specify the number of replica workers and even the max_concurrency per replica (how many requests each can handle in parallel, useful if the handler is async and can interleave work). Under the hood, Ray will ensure each replica (often pinned to a GPU) gets load-balanced traffic. If one GPU can handle more than one batch at a time (e.g., a lightweight model), you could run multiple replicas on the same GPU with fractional resource allotment, though typically for heavy models it’s one GPU per replica.

Ray Serve’s design makes it easy to scale out. In a large-scale text analytics pipeline, you might deploy N replicas of a text encoder model across a cluster, and Ray Serve will distribute the millions of incoming texts across those replicas for inference. It also integrates with Ray’s autoscaling, so the number of replicas can increase with load. Notably, Ray Serve has been used in conjunction with vLLM to handle LLM inference: Anyscale (the team behind Ray) demonstrated continuous batching on Ray Serve with vLLM, achieving huge throughput gains (up to 23× improvement on certain workloads) by combining system-level batching with vLLM’s internal optimizations (Achieve 23x LLM Inference Throughput & Reduce p50 Latency) . In essence, Ray Serve provides the serving skeleton (HTTP or RPC endpoint, routing, scaling, batching) while allowing specialized model runtimes like vLLM or Hugging Face TGI to do the heavy lifting inside. The benefit of Ray Serve is its generality and ecosystem integration – you can serve everything from simple scikit-learn models to massive transformers on GPUs using one framework, mixing and matching components (for example, a preprocessing deployment feeding into an LLM deployment). The overhead to consider is that it runs in Python and uses Ray’s messaging, which is very efficient but still not as bare-metal as something like Triton. Nonetheless, for many large-scale applications, Ray Serve strikes a balance between performance and development velocity, with proven support for micro-batching and concurrency in production (Maximizing Cost-Efficiency: Ray Serve for LLM Inference) .

NVIDIA Triton Inference Server

NVIDIA Triton is a high-performance inference server optimized for production deployment of AI models. It is framework-agnostic (supporting PyTorch, TensorFlow, ONNX, XGBoost, and more) and is implemented in C++ with support for REST/gRPC endpoints. Triton is built for maximal throughput – it has built-in scheduling algorithms to efficiently utilize GPUs. By default, Triton will automatically enable dynamic batching for any model that has a batch dimension. As long as you set a maximum batch size in the model’s config, Triton’s scheduler will combine incoming requests up to that batch size and feed them together to the model (Scaling ML Workloads Using Nvidia Triton Inference Server | QBurst Blog). Administrators can fine-tune this via config parameters: for example, setting a preferred_batch_size (to batch into specific sizes) or a max_queue_delay_microseconds to allow a tiny delay for gathering batches . This means that if you deploy a BERT-based classifier with max batch size 32, Triton can serve many users by packing their requests into 32-size batches whenever possible, achieving high throughput. Triton’s scheduler is smart enough to also consider multiple models or multiple instances: it can concurrently run different models on the same GPU if they are lightweight, or multiple copies of the same model for parallel execution . For example, you might run two instances of a small text classifier on one GPU, each handling its own batch – Triton will route requests accordingly and utilize the GPU’s capability to execute kernels from both instances in parallel (subject to GPU streaming multiprocessor availability). This concurrent execution feature is useful when serving ensembles or multi-stage pipelines too. Triton effectively acts as a traffic cop on the GPU, ensuring that at any given time, the GPU is executing a reasonably large batch from some model to avoid inefficient small runs .

One of Triton’s strengths is its optimized backends – for example, you can use TensorRT engines for neural networks to get extreme low latency, or use its FasterTransformer backend for optimized transformer inference. In the context of large-scale text inference, many companies use Triton to deploy transformer models for tasks like document classification or embedding extraction on GPUs. They benefit from its C++ efficiency and the fact that it handles multi-model scheduling, dynamic batching, and even HTTP/gRPC handling in one package. Deployment patterns often involve running Triton on each GPU server (with all models loaded), and load balancing across servers for scale-out. Horizontal scaling is achieved by simply adding more Triton instances behind a load balancer (since Triton itself is single-node, albeit handling multiple GPUs on that node) . In summary, Triton is a top choice when maximum throughput and utilization are needed – it automatically batches and schedules work to **reduce latency and increase throughput via better resource use **, but it requires writing a model config for each model and doesn’t allow custom Python logic in-process. It’s a pure inference serving engine focused on performance.

TorchServe

TorchServe is a model serving solution specifically for PyTorch models. It was jointly developed by Facebook and AWS and became an open-source project that many used for deploying PyTorch NLP and CV models. TorchServe operates with a notion of workers per model – each worker is a Python process that loads the model and processes requests. To handle batch inference, TorchServe provides a dynamic batching mechanism very similar to the others: you configure a maximum batchSize and an optional maxBatchDelay (in milliseconds) for each model in the server’s configuration (Batch Inference with TorchServe — PyTorch/Serve master documentation). The TorchServe frontend will accumulate incoming requests for up to that many items or until the delay is exceeded, then pass the batch to the model’s handler code . The model handler must be written to support a batch dimension (most of the built-in handlers like for image classification or text transformers already do). For example, if you set batch size 8 and max delay 50ms, TorchServe will group up to 8 text requests and infer them together; if only 4 arrived within 50ms, it will infer those 4 then proceed. This improves GPU utilization and throughput, as documented in AWS’s guides: batching in TorchServe “helps saturate the compute capacity and often leads to higher throughput” for inference (Optimize your inference jobs using dynamic batch inference with TorchServe on Amazon SageMaker | AWS Machine Learning Blog) .

TorchServe’s concurrency model is to scale via multiple workers. If you expect very high request volumes, you might start, say, 4 workers of a BERT model on a single GPU. Each will independently perform batching on its own queue of requests. In practice, running multiple workers on one GPU can increase throughput only if the model is small enough or if you need concurrency to handle multiple requests arriving exactly at the same time. Often a single worker with dynamic batching can already maximize a GPU, so additional workers mostly help on CPU-bound models or on CPU instances. TorchServe also supports multi-model hosting (you can serve several models on one instance, each with its own workers and batching settings). However, keep in mind that TorchServe is focused on PyTorch only; it won’t natively serve a TensorFlow model, for example. In recent years, the development of TorchServe has slowed (it entered maintenance mode), so while it’s still used in production, newer alternatives like Triton or custom FastAPI+PyTorch implementations have become more popular for cutting-edge use cases. Still, TorchServe remains a solid option especially when used via AWS SageMaker endpoints – AWS internally uses TorchServe for real-time PyTorch inference and supports the batch size and delay configuration through environment variables or the model server config . The main difference compared to Triton is that TorchServe runs everything in Python (with Java backend for the server), so it may not reach the same absolute throughput as Triton’s C++ engine for a single model on a single GPU. But it makes up for it with ease of use in the PyTorch ecosystem, and it provides straightforward batching and scaling knobs that practitioners can tune to meet their latency-throughput trade-offs.

Framework Comparison

Each of the above frameworks approaches large-batch text inference with a slightly different emphasis:

• vLLM: Specializes in large language model serving with continuous batching. It introduces iteration-level scheduling and memory optimizations (PagedAttention) to pack more concurrent text generation requests in GPU memory (Achieve 23x LLM Inference Throughput & Reduce p50 Latency). vLLM achieves extremely high throughput for LLM workloads, outperforming generic servers on those tasks, but it is narrowly focused on text generation use cases.

• Ray Serve: Provides a flexible distributed serving layer with built-in dynamic batching and Python-based concurrency. It’s ideal for composing inference pipelines (it can orchestrate pre/post-processing and multiple models). Ray Serve’s throughput gains come from its micro-batching (via asyncio) and ability to scale out across many machines easily (Dynamic Request Batching — Ray 2.44.1). The trade-off is a bit more overhead (running in the Python runtime, serialization costs) compared to a native server. It shines in scenarios requiring custom logic or serving many models, and can integrate with specialized backends (you can use Ray Serve to front-end vLLM or Triton, combining their strengths).

• NVIDIA Triton: Focuses on raw performance and efficient use of hardware. It’s a standalone server engineered in C++ that achieves low latency and high throughput through automatic batching and multi-model concurrency (Dynamic Batching & Concurrent Model Execution — NVIDIA Triton Inference Server) . It supports a wide range of model types and is commonly used when serving at “hyperscale” on GPUs. The downside is less flexibility for custom code – you mostly treat it as an optimized black-box inferencer. It requires model format compatibility (e.g. TorchScript or ONNX for PyTorch models) and more upfront configuration, but once running, it’s very robust and fast.

• TorchServe: Caters to PyTorch deployments with an easy configuration for batching and multiple workers (Batch Inference with TorchServe — PyTorch/Serve master documentation) . It is useful for teams heavily invested in PyTorch who want a ready-to-use solution. Compared to Triton, TorchServe is easier to plug into if your model is in Python (no need to convert model format), and it allows custom Python handlers. However, its performance is somewhat lower and it’s limited to the PyTorch framework. As of 2024, TorchServe is relatively mature but not adding new features, whereas Triton and Ray Serve are rapidly evolving with the latest optimizations.

In summary, vLLM (and similarly HuggingFace TGI) push the envelope for LLM inference throughput with continuous batching, Ray Serve offers a high-throughput distributed serving abstraction with micro-batching, Triton maximizes GPU efficiency through native batching and concurrency, and TorchServe provides batch inference capabilities in a PyTorch-centric package. The best choice depends on the use case: for example, log analysis with a BERT model might be well-served by Triton or TorchServe with dynamic batching, whereas a generative AI application might use vLLM or Ray Serve with continuous batching to handle thousands of tokens per second.

Conclusion

Efficient batch inference at scale requires carefully balancing batch size, concurrency, and hardware utilization. By leveraging high-throughput pipelines with dynamic batching and async scheduling, practitioners can process millions of text inputs with optimal throughput and cost-efficiency. Key techniques include buffering requests into micro-batches to exploit GPU parallelism, tuning batch sizes to saturate compute without exceeding latency or memory limits, and using concurrency (multiple workers or model instances) when appropriate to handle load. Modern inference frameworks implement these patterns out-of-the-box: solutions like vLLM, Ray Serve, NVIDIA Triton, and TorchServe each provide mechanisms to maximize throughput per GPU (Maximizing Cost-Efficiency: Ray Serve for LLM Inference). The engineering focus in 2024–2025 has been on minimizing overheads – from smarter batching algorithms to memory optimization – so that large-scale text inference can run orders of magnitude faster than naive one-by-one processing. By applying these state-of-the-art batching strategies and using the right tool for the job, engineering teams can deploy NLP models that handle millions of documents or streams in production while keeping latency within reasonable bounds and costs under control. All the techniques discussed are proven in production settings today, enabling high-throughput, scalable inference for cutting-edge text analytics applications.

Sources: High-throughput batching and continuous scheduling techniques (Achieve 23x LLM Inference Throughput & Reduce p50 Latency); Ray Serve dynamic batching and concurrency (Dynamic Request Batching — Ray 2.44.1) ; NVIDIA Triton batching and instance scheduling ; TorchServe batch config and usage (Batch Inference with TorchServe — PyTorch/Serve master documentation); vLLM memory optimizations and performance .