Technical description of OCR tokenization rules for historical n‑gram corpus construction

Technical description of OCR tokenization rules for historical n‑gram corpus construction The passage details internal text‑processing methods and tokenization edge cases. It contains no references to persons, institutions, financial flows, or misconduct, offering no investigative leads. Key insights: Describes preprocessing steps for OCR‑derived texts before n‑gram extraction.; Lists characters treated as separate tokens in the tokenizer.; Explains handling of hyphenated line breaks in scanned books.