The pipeline
- Normalize Unicode artifacts. Line endings are standardized. Optional rules remove soft hyphens and selected invisible spacing characters, convert nonbreaking spaces, and expand common PDF ligatures.
- Classify lines. The cleaner identifies blank rows, likely list items, short headings, blockquotes, indented code, and table-like rows.
- Estimate visual wrapping. Typical line width and nearby punctuation help identify line endings likely caused by page layout rather than meaning.
- Protect structure. Blank-line paragraphs and enabled structural categories are kept before any prose joins are attempted.
- Apply permitted repairs. The selected mode and rule toggles determine which boundaries can change.
- Build the proof ledger. Each transformation records its type, rule, before/after specimen, confidence level, and approximate source line.
Careful mode
Careful mode joins a boundary only when multiple signals point toward a visual wrap—for example, a wrap-sized line without sentence-ending punctuation followed by lowercase continuation text. Likely end-of-line split words may be repaired. Uncertain boundaries remain in the result and appear under “Needs review.”
Standard and Flatten
Standard mode accepts more probable wrap boundaries while still protecting enabled structure. Its wrap operations are labeled medium confidence. Flatten mode first applies enabled text-normalization rules, including conservative split-word repair, and then replaces the remaining line and paragraph breaks with spaces. It should be used only when document structure is unwanted.
Safety invariants
- Careful mode does not delete alphabetic or numeric content.
- Split-word repair removes only an end-of-line hyphen and the adjacent line break.
- Ligature normalization expands a typographic glyph to equivalent letters.
- Rule changes rebuild the result from the untouched source.
- Copy and download use the exact visible cleaned text.
- Change counts come from recorded operations; review and source-signal counts come from their corresponding detection records.
Known limitations
Copied text alone does not expose all PDF page geometry. The cleaner cannot reliably recover multi-column reading order, distinguish every legitimate compound from a split word, reconstruct tables, or remove repeated page headers when page boundaries are absent. Poetry, code, citations, legal clauses, and unusually formatted documents deserve particularly careful review.
Change record
Material changes to the cleaning rules or their documented limitations will be recorded on this page.