Batch Processing PDFs at Scale: Scripts, Queues and Cost Control

If you process more than 100 PDFs a day, you need a pipeline, not a UI. Here is a reference architecture that scales from 1 to 1 000 000.

The three laws of batch

1. Everything that can fail will fail. Network drops, 503s, out-of-memory errors. Build retries into every stage.
2. Idempotency wins. Use a deterministic job ID (sha256(input + options)) so retries don’t duplicate work.
3. Observe or regret. If you can’t see latency, throughput and failure cause per stage, you can’t fix regressions.

Reference pipeline

1. Ingest → S3 bucket + lifecycle rule (delete after 7 days).
2. Queue → SQS / RabbitMQ / Redis streams.
3. Worker → Lambda, Cloud Run or Kubernetes Jobs.
4. Store output → separate “outputs” bucket.
5. Index → Postgres or a search engine.
6. Notify → email / webhook when done.

Cost control

Our rule of thumb: a conversion worker costs about $0.0003 per CPU-second on commodity cloud. A standard PDF-to-Word job runs 2 seconds, so unit cost is $0.0006 or 60 cents per 1 000 jobs. If your bill looks higher, you’re either over-provisioned, OCR-heavy, or paying egress for giant files.

Autoscaling

Scale on queue depth, not CPU. CPU trails demand by minutes; queue depth is a leading indicator.

Backpressure

If downstream storage slows down, pause ingest at the API gateway rather than letting the queue balloon.

Monitoring that matters

• p95 end-to-end latency per tool.
• Error rate split by HTTP status.
• Queue depth and worker count.
• Cost per 1 000 jobs (daily).
• Age of oldest unprocessed job.

Using the PDFAPPLE API

Our REST API is designed for batch. Highlights:

• Signed-URL upload so you never ship bytes through our control plane.
• Webhook callbacks on completion.
• Idempotency keys as a header.
• Fine-grained rate limits you can see on the dashboard.

Full OpenAPI spec on the services page.

Monitoring sample

Here’s a Prometheus query for queue depth we actually use in production:

sum by (tool) (pdfapple_queue_depth{env="prod"})
  / avg by (tool) (pdfapple_worker_count{env="prod"})

Alert when it exceeds 5× for 10 minutes.

Open-source alternatives

If you prefer self-hosted, consider the stack ghostscript + poppler + tesseract + rabbitmq + python-rq. You will spend more engineering time but save on per-job cost at very high volume.

Design the pipeline once, feed it millions of files. Boring is good — boring never pages you at 3 am.

---

Tags: #batch PDF #PDF automation #PDF API #PDF processing at scale

---