Text Cleaner

The worst text bugs are the ones you cannot see. A product code copied out of a web page looks identical to the one in your database, the string comparison returns false, and there is nothing on screen to explain it. Usually the culprit is a character with no visual form: a zero-width space at U+200B, a byte-order mark at U+FEFF riding along at the front of a paste, a line separator at U+2028, or a soft hyphen sitting inside a word. Stripping that class of character is the first thing to try when two identical-looking strings refuse to match.

Word processors rewrite punctuation as you type, and the result travels wherever you paste it. Straight quotes become curly ones, a double hyphen becomes an em-dash, three periods become a single ellipsis character. In prose that is an improvement. In a config file, a shell command, or a snippet of code it is a syntax error, and the message points at a character that looks exactly like the one you expected. Text lifted out of a PDF is worse: ligatures such as fi arrive as a single glyph, with hyphenation left over from the original line breaks.

The non-breaking space at U+00A0 deserves its own warning. It renders as an ordinary space and behaves like nothing of the sort: an indentation-sensitive file — YAML, Python, a Makefile — breaks with an unexpected-indentation message while the line looks perfectly aligned. A pasted terminal command fails with command-not-found because the argument separator is not really whitespace. Converting every NBSP to a plain space fixes a whole family of these reports.

Line endings are the oldest of these problems and still the most common. Windows tools write CRLF, Unix tools write LF, and mixing them produces diffs where every line appears changed, shell scripts that fail with a bad interpreter error caused by an invisible carriage return after the shebang, and CSV parsers that leave a stray \r on the final field of each row. Normalizing to LF is the right default for anything that goes into git or a build pipeline; convert to CRLF only when a specific Windows consumer requires it.

Trailing whitespace and runaway blank lines are less dangerous but noisier. Spaces at the end of a line stay invisible until they show up as changed lines in a code review, and half a screen of empty lines in a data export is nothing but padding. Collapse runs of blank lines to one and trim line ends before you commit. Two caveats: Markdown uses two trailing spaces as a hard line break, and test fixtures may compare exact bytes.

Unicode normalization is the subtle one. The character é can be stored as a single code point, U+00E9, or as a plain e followed by a combining acute accent at U+0301. They look the same in every font and are not equal under a byte comparison, so a lookup fails, a deduplication misses, and a UNIQUE index accepts what should be a collision. macOS filesystems favour the decomposed form while most Windows and web sources produce the composed one, which is how a file list from one machine stops matching the same list from another. Normalizing to NFC before comparison solves it. NFKC goes further and is lossy — it rewrites fi to fi, ① to 1, and full-width characters to ASCII — so use it for search keys, not for data you store.

Cleaning before an import saves hours of database archaeology. A trailing space on an email address creates a second account for the same person. A tab inside a value shifts a whole CSV column. A field containing a comma or an embedded newline splits one row into two unless it is quoted correctly. Whitespace-only cells arrive as empty strings rather than NULL and quietly defeat every NOT NULL check. Run the steps in order: line endings, invisible characters, Unicode normalization, trimming, blank-line collapsing, and deduplication last — dedupe on unnormalized text misses the duplicates you were trying to remove.

Some text should never be cleaned. Base64 payloads, JWTs, private keys and hashes are exact byte sequences that a stripped newline or a case change destroys. In YAML, Markdown code fences and Python, leading whitespace is meaningful. If you have to verify a signature over a request body, keep the untouched original — even a CRLF to LF conversion changes the bytes the signature covers. Clean the human-entered lists and the pasted spreadsheet columns; leave machine-generated blobs exactly as they arrived.