Open source OCR with Vision Language Models: High throughput and low cost

Because we needed to OCR a little more than 100,000 pages to scale our graph-creation agent, we set out to find the most cost-effective way to do OCR with VLMs. In this post, we introduce an open source system that processes up to 20 pages per second while running on a single GPU. This translates to roughly $0.04 per 1,000 pages. For comparison, products like Azure's Document Intelligence or Mistral's OCR 4 charge $10 and $4 per 1,000 pages, respectively.

Model selection and system architecture

We pick GLM-OCR as our main model. It is openly available, weighs in at only 0.9B parameters, and is the best model on OmniDocBench, a popular PDF-parsing benchmark.

For best results, the model needs to be combined with a layout detection model. Instead of passing a full PDF page directly to the OCR model, the system runs through a multi-step process. The first step converts PDF pages to images. These images are passed to the layout model, which detects bounding boxes for page elements like headings, paragraphs, or tables. Next, the bounding boxes are used to create individual crops for each page region. The page regions are then passed individually to the OCR model for text recognition. The resulting text and region metadata can be assembled into a full Markdown or HTML reconstruction of the original document.

Benchmark setup

Z.ai provides an SDK that lets us run the whole workflow locally. We start from the SDK and benchmark throughput using an A100 GPU running on Modal. With maximum parallelization settings, we arrive at 5.29 pages per second.

We take apart the SDK and instrument it so that we can measure how long each step takes. We create a benchmark dataset of 100 PDFs from our full corpus. It consists of roughly 2,000 pages. We report wall-clock time spent on each step in the pipeline: reading files from cloud storage, PDF-to-image conversion, layout detection, cropping page regions, OCR, formatting, and writing outputs to cloud storage. We also report throughput in pages per second. Throughput is measured end-to-end including data uploads and downloads because that reflects how we use the system in production. Warm-up (downloading and initializing the models) is not included in the end-to-end measurement. Running the reassembled SDK with some tweaks and no optimizations, we process 4.51 pages per second.

Chasing high throughput

We notice that the system spends 46.8 percent of the time in layout detection. To make layout detection more efficient, we switch to a TensorRT inference backend. This brings down wall-clock time in layout to 7.1 percent, but the overall throughput of the system doesn't improve. Text recognition is now our bottleneck. We spend 78 percent of the time in the OCR stage.

The GPU utilization graph is very spiky. This means that we are not getting enough work onto the GPU fast enough. Looking through the vLLM logs, we notice that a lot of time is spent "rendering conversations". According to the vLLM docs, this step consists of preparing the prompts and images for inference. Conversation rendering runs on the CPU. After some more reading, we decide to scale out the number of API servers. vLLM uses a multi-process architecture: the core engine is responsible for GPU inference, while the API server handles requests and prepares data for inference. Scaling API servers means that more work can happen concurrently. We settle on scaling to 4 API servers after some benchmarking. This brings us to 5.56 pages per second, a sizeable improvement.

Reading through the technical report of the GLM-OCR model, we realize that the model supports speculative decoding. Speculative decoding uses a small draft model to predict the next N tokens, and the larger model verifies the tokens in a single forward pass. We apply speculative decoding with 3 tokens predicted by the draft model and measure again. Throughput rises to 7.14 pages per second.

Now, we turn to the data itself. Until now, we have passed page images at a resolution of 200 DPI to the layout model. We reduce that resolution to 100 DPI because fewer pixels for both models mean that they have less work to do. For our documents, there is no quality difference in the OCR outputs at the lower resolution. With all previous optimizations applied and the lower resolution, we get to 11.59 pages per second. This is a 2.5x improvement over our starting point.

We notice a few more opportunities for speedups. We switch from batching work per PDF to streaming page batches and micro-optimize some other parameters. With all the latest optimizations applied, we get to 14.26 pages per second on our benchmark dataset.

Up until now, we ran all experiments on a single A100 40GB using Modal as our GPU provider. Modal also offers the L40S at a slightly lower rate. It has similar VRAM, and our system should run on the L40S just as well. We rerun the last experiment on that card and achieve roughly the same throughput.

Production run

We're now ready for the production run: 116,000 pages across 2,000 documents. The run completes at 20 pages per second.

The overall Modal costs for this run amount to $4.64 (including CPU and memory). That's $0.04 per 1,000 pages. Our setup is 100 times cheaper than Mistral's OCR 4 and 250 times cheaper than Azure's Document Intelligence.

Find all code, including all optimizations, in this GitHub repository.