Walk me through steps of improving sentence transformer model used for embedding

Browse all previously published AI Tutorials here.

Table of Contents

• Improving Sentence Transformers for Document Digitization and Chunking

• Theoretical Advancements in Sentence Transformers 2024-2025

• Practical Strategies for Improving Embedding Quality

• Enhancing Retrieval Effectiveness and LLM Performance

• Optimization Techniques for Large-Scale Document Processing

• Key Trends and Conclusions

Improving Sentence Transformers for Document Digitization and Chunking

Document digitization involves converting large, unstructured documents into machine-readable text, then chunking that text for processing by embedding models and LLMs. Recent research (2024 and 2025) has explored ways to enhance sentence transformers to better handle this pipeline. Below, we review key advancements, practical strategies, retrieval techniques, and optimization methods that improve embedding quality for digitized documents.

Theoretical Advancements in Sentence Transformers (2024–2025)

Unified and Specialized Embedding Models: Researchers have developed new training paradigms to produce more powerful general-purpose text embeddings. For example, GTE (General Text Embedding) was trained with a multi-stage contrastive learning approach on a massive mixture of datasets ([2308.03281] Towards General Text Embeddings with Multi-stage Contrastive Learning](https://arxiv.org/abs/2308.03281#:~:text=,Furthermore%2C without)). Despite a modest model size (110M parameters), GTE-base outperforms OpenAI’s proprietary embedding model and even surpasses models 10× larger on the Massive Text Embedding Benchmark (MTEB) ([2308.03281] Towards General Text Embeddings with Multi-stage Contrastive Learning](https://arxiv.org/abs/2308.03281#:~:text=various NLP tasks into a,Furthermore%2C without)). This was achieved by significantly increasing training data in both unsupervised pre-training and supervised fine-tuning, yielding a broadly applicable embedding model ([2308.03281] Towards General Text Embeddings with Multi-stage Contrastive Learning](https://arxiv.org/abs/2308.03281#:~:text=substantial performance gains over existing,across various NLP and code)). Such results highlight that carefully curating diverse training data and tasks can boost embedding quality across domains without simply scaling model size.

Leveraging LLM-Generated Data: Another line of work examines using large language models (LLMs) to improve embeddings. Recent studies found that sentence embedding models trained on texts generated by LLMs can differ from those trained on human text, affecting embedding quality (Capturing the Relationship Between Sentence Triplets for LLM and Human-Generated Texts to Enhance Sentence Embeddings - ACL Anthology). To bridge this gap, one 2024 study introduced a novel loss function called Positive-Negative sample Augmentation (PNA) . PNA incorporates both human and LLM-generated sentence triplets during contrastive training, which was shown to mitigate the notorious embedding anisotropy problem (distributional collapse of embeddings) and improve semantic similarity scores (+1.47% Spearman correlation on STS tasks compared to a strong baseline) . This suggests that mixing diverse data sources and explicitly handling their differences can lead to more semantically uniform and accurate embedding spaces.

Generative Augmentation for Embeddings: A novel idea gaining traction is to use generative models at inference time to enrich sentence representations. Generatively Augmented Sentence Encoding (GASE) (Frank & Afli, 2024) proposes to create multiple synthetic variations of a sentence – such as paraphrases, summaries, or keyword extracts – using an LLM, and then average or pool their embeddings with the original (HERE). Unlike traditional data augmentation, GASE doesn’t require retraining the model; it trades a bit more inference compute for improved robustness. Experiments on STS benchmarks showed that feeding an encoder diverse paraphrased versions of an input adds semantic diversity and consistently improves similarity performance, especially for weaker base embedding models . This approach effectively injects additional context and wording diversity on-the-fly, boosting the encoder’s ability to capture meaning nuances without any parameter updates.

Bidirectional Context from Autoregressive Models: Since many modern LLMs (like GPT series) are autoregressive (one-directional), directly using them to produce sentence embeddings can be suboptimal – the first tokens may not encode later context. Very recent work (2025) addresses this by simply repeating the input text within the prompt to expose the model to its own output and capture bidirectional context in the embeddings (Retrieval Backward Attention without Additional Training: Enhance Embeddings of Large Language Models via Repetition.). By doubling or tripling the text (creating an “echo”), the model’s attention can incorporate future tokens, yielding more context-rich embeddings termed “Echo embeddings.” This Repetition with Backward Attention (ReBA) method improved understanding of the sentence start tokens, making autoregressive LLM embeddings closer in quality to those from bidirectional transformers . Notably, such techniques require no training – they exploit the transformer’s attention on repeated text to enhance semantic capture.

Practical Strategies for Improving Embedding Quality

Domain-Specific Fine-Tuning: In real-world document digitization, the content often comes from specialized domains (financial reports, legal contracts, medical scans, etc.). Fine-tuning or using domain-pretrained sentence transformers can significantly improve embedding relevance. For instance, in the financial domain, models like FinBERT (a BERT variant pre-trained on financial text) have shown superior performance on finance tasks (HERE). A 2024 study on financial report Q&A used a sentence transformer trained on 256 million question–answer pairs as the encoder . This model (Hugging Face’s multi-qa-mpnet-base-dot-v1) was chosen for its strong semantic search capability on Q&A data, reflecting how training on massive QA corpora yields embeddings well-suited for retrieval-based question answering. The takeaway is that aligning the embedding model’s training data with the target domain or task (through pre-training or fine-tuning) can greatly boost the quality of embeddings for digitized documents.

Chunk-Level Semantic Enhancement: When dealing with long documents, a practical challenge is deciding how to break text into chunks before embedding (since transformers have input length limits). Naïve strategies like fixed-length chunking can split sentences or sections in awkward places, reducing embedding quality. Research suggests using semantic or content-aware chunking to preserve coherence. For example, instead of blindly splitting every N tokens, one can split at natural boundaries (paragraphs, sections, or detected topics) so that each chunk is a self-contained unit of meaning. A study of long document QA in 2024 noted that fixed-length chunking is content-agnostic and often breaks cohesive sections, leading to incomplete context in each chunk (HERE). They propose ensuring no chunk spans multiple sections – essentially a content-aware approach that aligns chunks to the document’s structure . By reducing such “chunking errors,” the embedded chunks carry more complete information, which in turn improves retrieval and downstream answering accuracy. In practice, libraries for document processing (like Unstructured, LangChain, etc.) are beginning to support semantically aware splitting rules (e.g., keeping sentences or list items together) to improve the embeddings’ fidelity to the original document.

Multi-View Representations: An emerging strategy is to enrich each document chunk with multiple representations before indexing. Instead of relying on a single embedding per chunk, we can generate complementary embeddings that capture different aspects of the text. Multi-view Content-Aware Indexing (MC-indexing) is a 2024 approach that does exactly this: for each chunk (aligned to a section or logical unit), it creates three vectors – one for the raw text, one for a keyword summary, and one for an abstracted summary of that chunk . These multiple views increase the chance that at least one representation will closely match a given query’s wording. Notably, MC-indexing is a plug-and-play technique that requires no model fine-tuning; it can wrap around any existing sentence transformer and vector database. On a long-document QA benchmark, this method dramatically improved retrieval recall – for instance, achieving a 42.8% increase in top-1 recall on average (across various dense and sparse retrievers) compared to standard single-view chunking . By indexing the same content in diverse forms (original, keywords, summary), the system became far more effective at retrieving relevant chunks, highlighting a practical way to boost embeddings’ utility via data augmentation and index design.

Knowledge Distillation and Model Compression: Real-world deployments often face resource constraints, especially when embedding large-scale document collections on edge devices or with limited compute. A practical solution gaining popularity is distilling or training smaller, efficient embedding models that approximate the performance of large transformers. In late 2023, researchers demonstrated a recipe to train static embedding models (essentially lightweight models without heavy self-attention) that run 100× to 400× faster on CPU than typical transformers while retaining about 85% of the original embedding quality (Train 400x faster Static Embedding Models with Sentence Transformers) . These models, released via Sentence Transformers, enable on-device and real-time embedding of text, trading a small drop in accuracy for massive speed gains. Such techniques involve compressing knowledge from a large model into a small one (using clever training objectives and dimensionality reduction). For document digitization pipelines, this means embeddings can be generated and searched at scale (millions of documents) much more efficiently, making large-scale semantic indexing feasible in production environments.

Enhancing Retrieval Effectiveness and LLM Performance

Improving sentence transformers isn’t only about the embeddings themselves, but also how those embeddings are used in retrieval and how they feed into downstream LLM tasks (like question answering or summarization). Recent research emphasizes tight integration between chunking, embedding, and retrieval to maximize performance:

• Optimal Chunking for Q&A: The way documents are segmented can make or break a retrieval-augmented QA system. A 2024 study on financial report QA systematically evaluated chunking methods and found that an “element-based” chunking strategy (splitting documents by meaningful elements like sections, tables, and figures) yielded the best question-answering accuracy (HERE). By preserving the inherent structure of reports, the retrieved chunks contained more complete answers (e.g. entire table rows or section paragraphs), which allowed the LLM to generate correct and detailed answers. This outperformed baseline chunking by a significant margin, indicating that investing effort in smarter chunking directly improves LLM performance in document QA tasks . In practice, this suggests using document layout understanding (via OCR layout detection or templates) to guide chunking in digitization pipelines.

• Retrieval-Augmented Generation (RAG) Improvements: Once documents are chunked and embedded, retrieving the right chunks is crucial for LLMs to produce relevant answers. Beyond using better embeddings, new methods adjust the retrieval process itself. The MC-indexing approach mentioned earlier not only improved recall in evaluations but was shown to boost end-to-end QA accuracy due to more relevant context retrieved (HERE). Some research also explores when to retrieve – for instance, an adaptive RAG technique in 2024 analyzes the LLM’s input to decide if external retrieval is needed (HERE). While such methods go beyond embeddings, they underline a trend: effective document QA in 2024–2025 is achieved by holistically optimizing how text is chunked, embedded, and fed into LLMs. By enhancing each piece (better chunk coherence, higher-quality embeddings, multi-vector indexing, and intelligent retrieval logic), systems drastically reduce irrelevant or incomplete context, thereby improving the factual correctness and completeness of LLM-generated answers.

• Multi-Vector and Re-ranking Techniques: Another way to improve retrieval is to allow multiple embedding vectors per document or query, capturing different facets. We already saw multi-view chunk indexing for documents; similarly, queries can be expanded or re-encoded in multiple ways (e.g., using synonyms or context from the conversation) to increase recall. After initial retrieval, it’s also common to apply a cross-encoder re-ranker, which uses a full transformer to evaluate each retrieved chunk’s relevance more precisely. While re-rankers are outside the scope of sentence transformer models, they often use the same transformer architectures with fine-tuning. Recent systems (2024) combine a fast bi-encoder (for initial embedding search) with a powerful cross-encoder re-ranker, marrying speed and accuracy (Building LLM Applications: Sentence Transformers (Part 3) - Medium). The net effect is that improved sentence embeddings get relevant candidates in the ballpark, and then re-rankers (or the final LLM itself) ensure the very best chunks are selected for answer generation. This pipeline underscores that improving embeddings is synergistic with other retrieval components in boosting LLM performance on digitized documents.

Optimization Techniques for Large-Scale Document Processing

Dealing with enterprise-scale document repositories (millions of pages) requires optimization at multiple levels. Recent developments include:

• Approximate Nearest Neighbor (ANN) Indexing: Searching through tens of millions of embedding vectors in real-time is enabled by ANN algorithms. One popular choice is the HNSW (Hierarchical Navigable Small World) graph index, which significantly speeds up high-dimensional vector search while maintaining high recall (HERE). Modern vector databases (e.g. Weaviate, FAISS, Milvus) employ HNSW or similar ANN methods under the hood for sub-linear query time. For instance, the financial QA pipeline used Weaviate’s HNSW-based index to quickly retrieve relevant chunks from thousands of embedded SEC filings . In 2024, these ANN techniques are standard practice and continually being improved to handle ever-larger collections efficiently. Research from Google and others is producing refinements (like learned clustering or quantization strategies) to further accelerate vector search without sacrificing accuracy.

• Longer Context Embeddings: Another optimization is reducing the number of chunks by using embedding models with longer input limits. Traditional BERT-based models cap input around 512 tokens, but newer transformer architectures are pushing boundaries. For example, Alibaba’s GTE-XL (v1.5) models released in 2024 support inputs up to 8192 tokens (Alibaba-NLP/gte-large-en-v1.5 - Hugging Face). This means a single embedding can cover an entire long section or even a short document, avoiding the need to split it at all. By encoding more context into one vector, we minimize the loss of relationships between sentences that chunking might cause. In large-scale settings, fewer chunks per document directly translate to smaller indexes and faster retrieval. We also see LLM-based embedder approaches where an LLM with a 16k or 32k token window (like GPT-4-32k) could be used to embed very large texts; combined with tricks like ReBA (text repetition) to improve quality, this is a promising direction for handling lengthy digitized documents with minimal chunk fragmentation.

• Compression and Quantization: Storing millions of 768-dimensional float vectors is memory-intensive. Recent advances in vector compression help alleviate this. Techniques such as product quantization (PQ) and other compressive embeddings can shrink vector storage by an order of magnitude while preserving search accuracy. While much of this development predates 2024, the continued evolution of hardware and index structures keeps it relevant. Some 2025 pipelines report using mixed-precision or int8 quantized embeddings to fit larger corpora in memory without noticeable performance loss. Combined with efficient ANN, these allow scaling to web-scale document archives.

• Distributed and Streaming Processing: Though less discussed in papers, a practical optimization is distributing the embedding and indexing workload. In 2024, many frameworks (like Apache Lucene’s vector search or cloud-based vector DBs) enable sharding the index across multiple machines. This linear scalability ensures that even as organizations digitize every paper archive or scan thousands of books, the embedding-based search remains responsive. Streaming algorithms can ingest documents, chunk and embed them on the fly, updating the index in real-time. These engineering advances complement the research innovations, ensuring that state-of-the-art embedding techniques can be applied to truly large-scale document digitization projects.

Key Trends and Conclusions

In summary, the latest research (2024–2025) on sentence transformers for document digitization and chunking highlights a few clear trends:

• Smarter Training & Augmentation: Models trained on broader and more relevant data (e.g. GTE with diverse tasks, or combining human and LLM data) yield universally stronger embeddings. Additionally, creative use of LLMs – from generating paraphrases (GASE) to repeating inputs (ReBA) – can enhance embeddings without retraining, a practical boon for deployed systems.

• Structure-Preserving Chunking: There is a strong push toward chunking text in more intelligent ways. Content-aware and even “atomic” chunking (down to self-contained factoids (HERE)) help ensure that each embedded chunk is meaningful on its own. This reduces the chance of missing context and improves retrieval recall and precision. Both academic benchmarks and domain-specific evaluations confirm that better chunking directly improves downstream QA and retrieval-augmented generation performance (HERE) .

• Enhanced Retrieval Pipelines: Improving embeddings goes hand-in-hand with improving retrieval. Multi-vector representations (keywords, summaries) and hybrid search (dense + sparse) are being used to cover for any single model’s weaknesses. Meanwhile, integration of cross-encoders or LLM reasoning in the loop (to decide what to retrieve, or to rank results) is becoming common. The result is more relevant information fetched for LLMs, mitigating issues like hallucination by grounding answers in the retrieved text.

• Scalability and Efficiency: Finally, there is a clear recognition that methods must scale. Techniques like ANN indexing, model distillation for faster embeddings, and longer-context models are enabling the processing of larger document collections than ever before. These optimizations ensure that the theoretical improvements in embedding quality can actually be applied in real-world digitization pipelines under realistic latency and memory constraints.

By combining these advancements – from novel model training methods to practical chunking and indexing strategies – state-of-the-art systems in 2024 and 2025 are far more capable of digesting large, scanned or text-corpus documents and unlocking their information. High-quality sentence embeddings serve as the foundation of this pipeline, connecting raw digitized text to effective retrieval and accurate LLM understanding. The ongoing research suggests continued improvements on all these fronts, moving us closer to truly intelligent digital libraries and enterprise archives that can be searched and analyzed with human-like comprehension.

Sources:

• Frank, M. & Afli, H. (2024). Generatively Augmented Sentence Encoding. arXiv preprint arXiv:2411.04914 (HERE) .

• An, N. M., Waheed, S., & Thorne, J. (2024). Capturing the Relationship Between Sentence Triplets for LLM and Human-Generated Texts to Enhance Sentence Embeddings. Findings of EACL 2024 (Capturing the Relationship Between Sentence Triplets for LLM and Human-Generated Texts to Enhance Sentence Embeddings - ACL Anthology) .

• Li, Z. et al. (2023). Towards General Text Embeddings with Multi-stage Contrastive Learning (GTE). arXiv:2308.03281 ([2308.03281] Towards General Text Embeddings with Multi-stage Contrastive Learning](https://arxiv.org/abs/2308.03281#:~:text=various NLP tasks into a,Furthermore%2C without)) ([2308.03281] Towards General Text Embeddings with Multi-stage Contrastive Learning](https://arxiv.org/abs/2308.03281#:~:text=substantial performance gains over existing,across various NLP and code)).

• Dong, K. et al. (2024). MC-indexing: Effective Long Document Retrieval via Multi-view Content-aware Indexing. Findings of EMNLP 2024 (HERE) .

• Jimeno-Yepes, A. et al. (2024). Financial Report Chunking for Effective Retrieval Augmented Generation. arXiv:2402.05131 (HERE) .

• Jelassi, S. et al. (2025). Retrieval Backward Attention: Enhance Embeddings of Autoregressive LLMs via Repetition. arXiv:2502.20726 (Retrieval Backward Attention without Additional Training: Enhance Embeddings of Large Language Models via Repetition.) .

• Reimers, N. & Aarsen, T. (2023). Training 400x Faster Static Embedding Models. HuggingFace Blog (Train 400x faster Static Embedding Models with Sentence Transformers) .

• Raina, V. & Gales, M. (2024). Question-based Retrieval using Atomic Units for RAG. (Referenced in Dong et al. 2024) .