Optimizing Enterprise Retrieval-Augmented Generation Through Sequential Document Processing and Deterministic Dispatching

In the rapidly evolving landscape of artificial intelligence, enterprise-grade Retrieval-Augmented Generation (RAG) systems are facing a critical bottleneck: the inefficient use of computational resources. As organizations transition from pilot programs to production-scale deployments, the cost of processing vast amounts of data through Large Language Models (LLMs) has become a primary concern for Chief Technology Officers and data architects. A new methodology, emerging from the "Enterprise Document Intelligence" framework, proposes a shift from the traditional "batch" processing of retrieved data to a "sequential" model. This approach, centered on a deterministic dispatching mechanism, promises to reduce token costs by as much as 80% for specific classes of queries while maintaining the high accuracy required for professional environments.

The core problem lies in the "naive" RAG baseline that most developers currently employ. In a standard pipeline, when a user asks a question—such as "What is the effective date of this policy?"—the system retrieves the top-K most relevant document chunks, usually five or ten. It then feeds all these chunks into the LLM simultaneously. While this "batch" method is effective for complex tasks like comparing different sections of a contract, it is remarkably wasteful for simple factual lookups. Often, the very first chunk retrieved contains the complete answer. By forcing the LLM to read the subsequent four or nine chunks—which may consist of irrelevant signatures, footnotes, or historical data—companies are essentially paying for redundant computation.

The Two Regimes of Generation: Batch vs. Sequential

To address this inefficiency, architectural experts are now distinguishing between two primary regimes for feeding retrieved candidates into the generation brick of a RAG system.

The first is the fixed top-K pipeline, which remains the default in most tutorials and open-source frameworks. In this model, the token cost is a direct function of the question length plus the total volume of all K chunks. Regardless of whether the answer is found in the first sentence or the last, the cost remains static and high. For an enterprise managing a corpus of 50,000 policies, these "rounding errors" in token usage accumulate into significant monthly expenditures.

The second regime is "Sequential Feeding." This method treats the retrieved candidates as an ordered list. The system feeds the top-ranked candidate to the LLM and asks a specific question regarding its sufficiency. If the LLM determines that the answer is present and complete, the process stops immediately. If not, the system "escalates" to the second candidate, and so on. This "top-1 first" strategy ensures that the system only consumes the resources necessary to satisfy the query.

The Role of the Typed Sufficiency Signal

The success of the sequential regime depends entirely on the LLM’s ability to self-report the quality of the information it has processed. This is achieved through what is known as a "typed contract." Within the generation brick, the system utilizes a schema—often a Pydantic model in Python—labeled AnswerWithEvidence.

This schema includes two critical boolean fields: answer_found and complete_answer_found.

1. answer_found: Indicates if the candidate chunk contained the specific information requested.
2. complete_answer_found: Indicates if the information provided a total answer rather than just a fragment.

By using these booleans instead of a volatile "confidence score," the system creates a deterministic loop. If both fields are true, the loop exits. If the answer is found but is incomplete, the system continues to the next candidate to find the missing pieces. If nothing is found, it proceeds until the K-limit is reached. This structured feedback loop prevents the "agentic temptation" where an LLM might otherwise wander through a document indefinitely, burning tokens without a clear objective.

Chronology of RAG Evolution

The development of these two regimes marks a significant milestone in the timeline of document intelligence:

• Phase 1: Naive RAG (2022–Early 2023): Systems focused on basic vector embeddings and similarity searches. The "all-at-once" batch approach was the only standard, as the primary goal was simply getting the LLM to see the right data.
• Phase 2: Advanced RAG and Reranking (Late 2023): Developers introduced reranking models to ensure the most relevant chunks were at the top of the list. However, even with better ranking, systems still processed all top-K chunks simultaneously.
• Phase 3: The Modular Enterprise Framework (2024): The industry began treating RAG as a series of distinct "bricks"—parsing, question parsing, retrieval, and generation. This modularity allowed for the "sequential" vs. "batch" decision to be made at the architectural level.
• Phase 4: Deterministic Dispatching (Present): The current frontier involves using a question parser to route queries to the most efficient processing regime before the LLM is even engaged for the final answer.

Supporting Data: Cost and Efficiency Analysis

The financial implications of switching to a sequential model are substantial. In a typical enterprise workload, such as insurance claims processing or legal discovery, the distribution of questions usually leans toward factual lookups.

Data collected from back-of-the-envelope calculations for an enterprise insurance Q&A workload reveals the following:

• Factual Lookups (Approx. 80% of traffic): Using sequential processing results in an average token reduction of 80% per query, as the top-1 or top-2 chunks usually suffice.
• Complex Queries (Approx. 20% of traffic): For listings, comparisons, or summaries, the system defaults to batch processing. While this pays the full token cost, it is only applied when necessary.

On a batch of 100 questions, a standard pipeline might consume 500 units of context (100 questions x 5 chunks). A sequential-enabled pipeline, however, might only consume 175 units (80 questions x 1.25 chunks average + 20 questions x 5 chunks). This represents a roughly 65% total saving on input tokens for the generation phase.

The Dispatch Decision: Routing by Intent

A critical component of this architecture is the "Dispatcher." Instead of picking one regime for the entire pipeline, the system evaluates each question individually. This is not an "agentic" decision made by a flexible LLM, which can be unpredictable and difficult to audit. Instead, it is a "deterministic dispatch" based on the metadata generated during the question-parsing phase.

The dispatcher reads two primary attributes: answer_shape and decomposition.

• If the answer_shape is a single value (a date, an amount, or a boolean), the dispatcher routes it to the Sequential path.
• If the answer_shape is a list, a comparison, or a summary, it is routed to the Batch path.

This logic holds true across various sectors. In the legal field, a question about a "Termination Notice Period" is a sequential factual lookup. A request to "Summarize all indemnification exclusions" is a batch task. In the medical field, "What is the patient’s blood type?" is sequential, while "Compare the last three lab results" is batch. By anchoring the decision in the shape of the expected answer, the system maintains a clear audit trail—a prerequisite for compliance in regulated industries.

Professional Analysis and Broader Impact

The move toward sequential processing reflects a broader trend in AI: the shift from "brute force" computation to "intelligent orchestration." As the initial hype around LLMs matures into practical application, the focus is shifting toward sustainability and precision.

The "Amplify the Expert" philosophy, which underpins this research, suggests that AI should not replace the human expert but rather provide a highly efficient toolset that mirrors professional workflows. An expert human researcher does not read an entire library if the answer is on the first page of the first book they open; they stop once they have verified the information. Sequential RAG brings this human-like efficiency to automated systems.

Furthermore, this development has significant implications for latency. While sequential processing might involve multiple LLM calls in the worst-case scenario (where the answer is in the 5th chunk), it drastically reduces the "Time to First Token" and overall processing time for the majority of queries. In a world where user experience is tied to responsiveness, the ability to return a factual answer in a fraction of the time required for a full batch process is a competitive advantage.

Conclusion and Future Outlook

The distinction between sequential and batch feeding of retrieved candidates is more than a technical optimization; it is a fundamental refinement of the RAG architecture. By leveraging a typed contract and a deterministic dispatcher, enterprise systems can achieve a level of efficiency that was previously unattainable.

As document corpuses continue to grow into the millions of pages, the cost of "naive" processing will become unsustainable. The industry is likely to see a broader adoption of these "stop-when-sufficient" loops. Future iterations may include even more granular dispatching rules, perhaps incorporating metadata about document reliability or historical retrieval performance. For now, the integration of the sequential regime stands as a vital step in transforming RAG from a promising experiment into a robust, cost-effective enterprise utility. The decision-making power, once left to the "black box" of the LLM, is being reclaimed by the parser, ensuring that every token spent is a token required.