Surging RAG: Processing 17,000 Pages in 20 Minutes Without Breaking the Bank

1. Executive Summary

We recently built a production-grade AI document assistant on InMotion Cloud, then pushed it to its absolute limits with an aggressive real-world deadline. This is a shorter version of our benchmark; if you want the full report with all 15 measurement tables, per-step VM startup timings, and rejected design alternatives, refer to the full technical report.

• The Challenge: A customer uploads a 17,001-page archive and expects full searchability within 20 minutes.
• The Bottleneck: Three baseline OCR workers fell severely behind schedule—left alone, they would need about 47 minutes.
• Autonomous Scaling: A lightweight watcher detected the delay within seconds and queried the InMotion Cloud OpenStack API to auto-provision virtual machines up to a 9-worker cap.
• Rapid Onboarding: New instances joined the Kubernetes cluster and began processing pages roughly 88 seconds after the scaling decision.
• Success: All 17,001 pages were fully indexed 67 seconds ahead of the 20-minute deadline.
• Accurate RAG Output: Post-ingestion, the assistant provided exact answers complete with verifiable PubMed Central citations.

During the burst, the Fixed Cost usage metric climbed from $169.53 to $430.56 (~8% over the $399 plan allowance), shifting platform health status from Good to Bursting. Despite exceeding plan limits, the monthly bill remained fixed at $399 with zero overages, throttling, or capped performance.

2. The Scenario and Control Test

Enterprise Retrieval-Augmented Generation (RAG) platforms rarely fail on chat traffic—they stall when a customer uploads a massive back-catalogue of scanned contracts, medical papers, or reports and expects them to be searchable right away.

In our benchmark, the uploaded archive contained 17,001 pages (14.8 GB) drawn from commercially licensed PubMed Central (PMC) Open Access papers. About 70% of the archive consisted of scanned PDFs and TIFF images requiring optical character recognition (OCR), while the remaining 30% comprised native PDFs, Word documents, and plain text.

OCR is CPU-bound: each 8-vCPU worker processes about 2.0 pages per second at 85–90% CPU utilization. With uploads streaming into Postgres at 41 pages per second, three baseline workers would take 47 minutes—more than double the 20-minute promise.

To prove the archive was genuinely read and indexed during the run, we verified every test with a control question whose answer exists only in the newly uploaded papers:

"In critically ill older patients starting enteral feeding, in what share of patients was serum asprosin high on the first day?"

Before the upload, the assistant replied "I don't have anything on that." After the 19-minute ingestion run, the assistant answered 96% and cited [PMC10000515].

3. Architecture at a Glance

The cloud project runs on a single private network (10.20.0.0/24) with five standing m7i.medium machines and an elastic pool of up to nine m7i.2xlarge OCR workers:

The system separates the workload into two distinct paths:

• Ingestion (the heavy path): Pages stream into a shared Postgres 17 + pgvector queue on surge-vector-db. Each worker node runs an ingestion pod via a Kubernetes DaemonSet with 8 single-threaded Tesseract OCR processes (one per vCPU) and a local bge-small-en-v1.5 embedding model baked into the container image. Workers claim pages using row-level locks (SKIP LOCKED) with expiring leases, split text into deterministic chunks, embed them, and write vectors back to Postgres.
• Answering (the light path): When a user asks a question in Open WebUI (connected through a restricted two-hop SSH tunnel via surge-bastion), Nous Research Hermes Agent (v0.21.5) calls a single MCP tool, search_documents. All 17 built-in Hermes toolsets and session memory are disabled so the agent can only answer from the database. Candidate passages from pgvector are rescored on CPU by a two-stage reranker (TinyBERT followed by MiniLM-L-12), and a groundedness guard verifies that every number in the final response appears in the retrieved text before returning the cited answer.

4. Autonomous Scaling and Fixed Cost Mechanics

Rather than waiting for the queue to overflow, the scaling watcher on the operator workstation continuously projects the completion time against the fixed 20-minute deadline—counting both active workers and VMs still booting.

When the watcher projects a missed deadline, it calls the OpenStack Nova API (POST /servers, spaced at least 15 seconds apart) after verifying hard project caps (maximum 9 workers, 14 total instances, 84 vCPU, 312 GB RAM, and a $478.80 abort ceiling). Each new VM boots Ubuntu 24.04 LTS, receives the preloaded worker container image over SSH, joins the k3s cluster in the background, verifies database reachability, and starts claiming pages in about 88 seconds.

On InMotion Cloud's Fixed Cost Advantage plan ($399/month for 24 vCPU, 82 GB RAM, and 2,000 GB local NVMe), the usage figure reflects allocated resources right now—not CPU utilization and not an hourly bill:

Nothing charges extra, throttles CPU, or blocks provisioning when the project enters Bursting. Most importantly, there is no automatic scale-down: you can keep the six added worker VMs running while you decide whether the workload surge was a one-off event or your new baseline.

5. Keeping Customer Data Inside the Project

A managed cloud OCR service would have offloaded the CPU spike, but it would have required sending every page of the customer's private archive to a third party and paying per-page fees that grow with every upload. Keeping OCR, embedding, vector search, and reranking inside the project boundary ensures strict data locality:

The hosted language model (claude-haiku-4-5) never sees the document archive, page images, or vector index—it receives only the user's question and the few reranked passages needed to write the cited answer. Because the model adapter also includes a generic OpenAI-compatible driver, teams with strict zero-egress requirements or GPU-backed instances can swap in a local model (vLLM, Ollama, or llama.cpp) with a single configuration change.

6. Benchmark Results and Key Lessons Learned

In our unattended reference run, the system scaled from 3 to 9 workers and drained all 17,001 pages in 18 minutes and 53 seconds—finishing 67 seconds ahead of the 20-minute deadline:

Across a 19-question evaluation suite (single-fact, multi-paper, and unanswerable trap questions, run three times each), the assistant scored 26 of 26 on answerable parts with verified PMC citations, 12 of 12 on refusing trap questions, and 0 hallucinated numbers.

Six engineering lessons stood out from building and testing on live machines:

1. Pin one thread per OCR process. Allowing 8 OCR processes to spawn multi-threaded workers across the same 8 cores caused severe lock contention and ran 148 times slower than 8 single-threaded processes.
2. Put the work queue in the shared database. Per-node SQLite queues left newly joined VMs idle while older nodes were swamped. A shared Postgres queue with SKIP LOCKED leases let new workers pull pages the second their pod started.
3. Flush upload batches by byte size, not just page count. Fifty uncompressed TIFF scans (~55 MB) in a single batch crashed a database worker; flushing at an 8 MB ceiling kept memory flat at 35.8 MB/s.
4. Return full passages and enforce a groundedness guard. Short 400-character snippets caused the LLM to guess missing numbers beside real citations. Returning full passages and rejecting any reply containing unverified numbers eliminated hallucinations.
5. Run node joins in the background. Waiting for each VM to finish joining k3s before launching the next caused an earlier run to miss the deadline by 70 seconds.
6. Verify service reachability before marking a node ready. A Kubernetes node can report Ready before kube-proxy finishes syncing rules; testing cluster DNS and Postgres connectivity during join caught and self-healed a silent networking stall.

7. Conclusion

A 20-minute compute spike is cheap on any cloud; the real operational trap on pay-as-you-go infrastructure is the pressure to tear capacity down immediately afterward—or pay thousands of dollars a month if you leave the extra instances running.

With Fixed Cost Advantage on InMotion Cloud, our RAG cluster burst from 3 to 9 workers, absorbed a 17,001-page OCR surge with over a minute to spare, and left all nine workers running at the same $399 flat monthly fee.

Want the complete breakdown—including the assistant retrieval flowchart, CPU benchmarks for local Gemma 3 models, rejected architecture alternatives, and the full glossary? Read the Surging RAG Full Technical Report.