Logistics companies deal with a relentless flow of structured-but-messy documents: bills of lading, customs declarations, packing lists, carrier invoices, proof-of-delivery scans. For a mid-sized freight forwarder, a single shipment might touch eight to twelve distinct document types before it closes. When we first looked at this problem for a client, the instinct was to feed everything into an LLM and call it done. That instinct was wrong—or at least, premature.
Here is what we actually built, what broke, and what is working now.
The Starting Problem
The client's operations team was spending roughly four hours per day per person on data entry: pulling figures from PDF invoices, matching them to purchase orders in their transport management system, flagging discrepancies, and forwarding exceptions to finance. Documents arrived from dozens of carriers in varying formats—some machine-readable PDFs, many scanned images of paper originals, a handful of Excel attachments dressed up as structured data.
The goal was not to eliminate the operations team. It was to redirect their time from copy-paste work to exception handling, supplier negotiations, and customer communication—work that actually requires judgment.
Why a Pure LLM Approach Failed First
Our first prototype fed raw PDF text to a large language model with a detailed extraction prompt. The model was asked to return a JSON object with fields like invoice_number, carrier_name, total_amount, line_items, and payment_terms.
On clean, machine-readable PDFs from major carriers, field-level accuracy was around 92 percent. Acceptable for a demo, not acceptable for finance. On scanned documents with skewed alignment, faded ink, or non-Latin characters from Eastern European carriers, accuracy dropped into the sixties. Worse, the model hallucinated invoice numbers when scan quality was poor enough that it could not read the actual digits. That is the failure mode that matters: not I don't know, but I'll guess confidently.
We also ran into a token cost problem. Feeding full PDF text—including three pages of boilerplate payment terms—for every document was expensive and slow at volume.
What We Rebuilt
The architecture we settled on has three stages.
Stage one: classification. Before any LLM call, a lightweight classifier—fine-tuned on roughly 2,000 labeled examples from the client's own archive—identifies the document type and the issuing carrier. This lets us apply document-specific extraction templates and skip irrelevant sections. A Lufthansa Cargo air waybill has a predictable structure; knowing that upfront reduces the extraction problem significantly.
Stage two: OCR with confidence scoring. For scanned documents, we run dedicated OCR (we used Google Cloud Document AI for this client, though AWS Textract is a reasonable alternative) and retain per-word confidence scores. Anything below a threshold is flagged for human review before LLM extraction is attempted. The model never sees low-confidence text—it receives a cleaned, structured representation of the document.
Stage three: structured extraction with validation. The LLM—GPT-4o-mini, chosen for cost and latency—receives a trimmed document representation and a strict JSON schema. Its output is then validated against business rules: does the currency match the carrier's typical billing currency? Does the total reconcile with line items? Is the invoice date within a plausible range? Failures route to a human review queue, not directly to the TMS.
This is more engineering than just use an LLM, and it is. But the multi-stage approach brought accuracy on clean documents above 98 percent and reduced the volume of exceptions requiring human review by roughly 70 percent compared to the naive prototype. These are our internal measurements for this engagement; results vary with document quality and supplier mix.
The Human Review Queue Is Not a Failure Mode
One framing shift that helped the project land internally: we stopped calling the review queue a fallback and started calling it the intended output for ambiguous cases. The system's job is not to process every document automatically. It is to process confidently-processable documents automatically, and to present ambiguous documents to a human in a way that makes review fast.
The review interface shows the original document side-by-side with the extracted fields, highlights the specific field that triggered the review, and pre-populates everything the model was confident about. A reviewer can confirm or correct a flagged invoice in under ninety seconds—compared to a four-minute full manual entry for every document before automation.
What We Chose Not to Automate
Two document categories stayed entirely manual.
First, dispute documents—emails or PDFs from carriers contesting a chargeback or requesting rate amendments. These require judgment about contract terms, relationship history, and escalation risk. Automating extraction here would save little time and create false confidence where none is warranted.
Second, regulatory compliance fields in customs declarations. The client's customs broker has direct liability exposure on these entries; they wanted a human signature on every field. Respecting that boundary was the right call, and it was not our place to argue otherwise.
Cost and Change Management
The infrastructure cost for this system—OCR, LLM calls, compute—runs to roughly €0.04 per document at current volume. At 800 documents per day, that is approximately €1,000 per month. The manual processing cost before automation was roughly six person-hours per day, which is more easily framed as opportunity cost than direct savings, but the order of magnitude favors automation clearly.
The harder cost was change management. The operations team was skeptical—reasonably so, after watching the first prototype make confident errors. We ran a two-week parallel period in which the system's outputs were checked against manual results before anyone trusted the automated queue. That parallel period was not a nice-to-have; it was what built the trust needed to actually change workflows.
Questions Worth Asking Before You Build
If you are evaluating a similar project, the questions that matter before writing any code are:
- What is the acceptable error rate, and who owns the error when it happens?
- How variable is your document corpus, and do you have enough labeled examples to bootstrap a classifier?
- What does the human review interface look like, and is it fast enough that reviewers will use it without friction?
- Are there document categories—regulatory, legal, high-liability—that should stay manual regardless of automation accuracy?
The technology is not the hard part. The hard part is building a system that your operations team will trust enough to change how they work. Get the parallel validation period, design the review interface before the extraction pipeline, and be willing to draw clear boundaries around what the system should not touch.