Beyond Character Error Rate: Evaluating Diacritic Fidelity and Post-Processing Effects in Classical Arabic OCR

Abstract

Primary results.

The CER rise on the unvowelled and al-Jami' scans comes from one page on which v0.1 repeats digits to the token limit (page CER 6.5). With one retry at repetition penalty 1.1, mean CER on the verified unvowelled set is 0.083. The "all 300 dpi pages" aggregate is a 10-page 300 dpi sample plus the 49 Inna Kunna pages, which overlap the unvowelled set. Full per-metric values, including invented and omitted marks, the retry variant and the second training run: evidence/primary-results.csv.

How the numbers are computed.

Letters-only CER

Levenshtein distance divided by reference length, over a text view with the diacritics, Qur'anic annotation marks and tatweel removed (Unicode NFC, no letter folding). It is unbounded above 1 when an output loops.

Conditional DER

The two texts are split into letters with their attached marks and aligned. Among aligned Arabic letters that carry a mark on at least one side, DER is the share whose marks differ. Letters the alignment does not match are excluded from both numerator and denominator, so DER measures marks conditional on correct letters.

Why DER can be undefined

If no aligned letter carries a mark on either side, as with an empty output or an unvowelled page read without vowels, the denominator is zero and DER is undefined. Such pages leave the DER mean and the paired DER comparison. That is why the DER page counts above can be smaller than the CER counts. On an unvowelled reference, page DER is 1.0 if any mark is invented, so pooled invented-mark rates are reported alongside.

Invented and omitted marks

An aligned letter that is bare in the reference but marked in the output is invented; one marked in the reference but bare in the output is omitted. Both are pooled per 1,000 aligned letters.

Alignment assumptions

An optimal Levenshtein alignment over base characters; mark order on a letter is ignored. Substituted, inserted or deleted letters carry no DER contribution, so text loss can lower DER's denominator without penalty. It is counted instead by CER and by omitted-word runs.

Quotation layer

Spans between ﴿…﴾ of at least eight letters are matched against the Qur'an after normalising letters. A window of one to four verses with a score of at least 90 replaces the span with the canonical wording. The layer rewrites only the model output.

Intervals

Paired bootstrap over pages: 4,000 resamples, seed 1, percentile 95% interval of the mean per-page difference, over pages where the metric is defined for both systems. No multiple-comparison correction. The intervals describe variation across the evaluated pages only; they do not estimate uncertainty about other books, editions, typefaces or publishers.

Why the marks matter

Principal claims and what supports them.

Downloadable evidence

• evidence/claims.csv — the claims above, one per row
• evidence/primary-results.csv — raw-output results by evaluation set, system and metric, with page counts and intervals
• evidence/postprocessing-results.csv — DER with the quotation layer off and on
• evidence/release-gate-verdicts.csv — release-gate verdicts with the layer off and on
• evidence/experiment-manifest.json — systems, revisions, datasets, metric definitions, provenance and gaps
• evidence/README.md — what each file contains and how to read it

All aggregate values were exported by script from the audit's recomputed results; no page text or page image is included.

Read the results with these in mind.

• Narrow book and edition coverage. The typeset evidence is one edition series; the human-verified low-resolution evidence is one 44-page book, which also holds every partially or unvowelled test page.
• Reference quality varies. Some scan results rest on silver references produced by an LLM-based OCR pipeline and never checked by a person. A visual check suggests some references in the verified low-resolution book may disagree with the scan, in an unknown direction.
• Conditional metric blind spots. DER ignores letters the alignment does not match, is undefined on some pages, and carries little information on unvowelled references; CER is unbounded and page-averaged, so a single looping page can dominate a set mean.
• Distribution shift. All real training pages are typeset and fully vowelled; the scanned and partially vowelled evaluation pages differ from them.
• Confounded training comparison. The two training runs differ in many variables at once, and each ran once with an unlogged seed; the comparison supports no causal claim about synthetic data.
• Checkpoint selection. The checkpoint was chosen on 36 validation pages, which are reported separately and not used as independent evidence.
• Inference variability. Running the same model twice changed 28 of 157 answers; the cause was not recorded.
• Post-hoc release-gate change. The project's release gate was revised after v0.1 failed its first version; the same outputs then passed. The pass/fail line between the two runs is within bootstrap-seed noise.
• Data release. The scanned documents and reference transcriptions cannot be released, which limits independent reproduction of the scan results.

The role of generative AI.

How to cite.

Qudrath, M. F. (2026). Beyond Character Error Rate: Evaluating Diacritic Fidelity and Post-Processing Effects in Classical Arabic OCR (Version 1.0) [Preprint]. Zenodo. https://doi.org/10.5281/zenodo.23683237

@misc{qudrath2026beyond,
  author       = {Qudrath, Mohammed Fazalullah},
  title        = {Beyond Character Error Rate: Evaluating Diacritic Fidelity and Post-Processing Effects in Classical {A}rabic {OCR}},
  year         = {2026},
  publisher    = {Zenodo},
  version      = {1.0},
  doi          = {10.5281/zenodo.23683237},
  url          = {https://doi.org/10.5281/zenodo.23683237},
  note         = {Preprint. Research dossier: https://pandanlabs.com/research/arabic-ocr-diacritics/}
}

Download citation.bib. Cite the version DOI for this exact text; 10.5281/zenodo.23683236 always resolves to the latest version.