Inside the TokenWorks Document Intelligence Pipeline
A look at the public OCR pipeline’s page-level extraction, diagnostics, OCR routing, post-processing, evaluation, and structured outputs.
TokenWorks Editorial · TokenWorks LLC
Explanatory architecture diagram; not a processing result. Image credit: TokenWorks; original explanatory architecture diagram, not an OCR run or result.
Inspect the page before choosing a route
The public TokenWorks OCR Pipeline processes PDF documents at page level. It inspects the existing text layer and uses diagnostics to inform extraction or OCR routing. The maintained implementation exposes a page-text runner alongside the compatibility entry point.
Keep the output inspectable
The repository describes diagnostics, routing, cleanup, and quality artifacts alongside text extraction. Page text can be emitted in CSV and JSON, with separate files for processing progress, diagnostics, layout regions, and quality information. Raw OCR can be retained for review.
Evaluation is part of the work
The engineering effort includes evaluation and structured document modeling. UC Berkeley Data Discovery’s Spring 2026 program separately documents a TokenWorks OCR evaluation project by Vedashnii Raghu, Tiantong Lu, Rithik Anumula, and Jerry Cheng, mentored by Adam Anderson. This is a program research collaboration, not a university-wide corporate endorsement.
Open engineering, bounded claims
The repository is published under Apache-2.0 and links to its current runbooks and technical documentation. It does not establish a universal OCR accuracy result or prove readiness for every document collection. This update describes source-code capabilities, not the availability of a hosted OCR service. Consult the repository for installation requirements, supported routes, and evaluation guidance.
Sources and project links
https://github.com/TokenWorks-LLC/OCR_pipeline