Where messy text comes from, and which switches fix it
Most dirty text arrives from one of a handful of sources, and each leaves its own marks:
- A PDF. Every visual line becomes a line break, and words may be split by a hyphen at the end of a line. Use Remove line breaks with Keep paragraph breaks on, then Collapse repeated spaces.
- A web page. Pages bring non breaking spaces, curly quotes and sometimes whole tags. Use Strip HTML tags, Replace unusual spaces and Straighten curly quotes.
- An email. Hard wrapped lines, indentation from quoting and trailing spaces. Use Trim each line and Remove line breaks.
- A spreadsheet or export. Trailing spaces that make lookups fail, and empty rows. Use Trim each line and Remove empty lines.
- A chat window or an editor. Zero width spaces and byte order marks that cause matches to fail for no visible reason. Use Remove invisible characters.
What each switch does
- Trim each line removes spaces and tabs at the start and end of every line.
- Collapse repeated spaces turns any run of spaces and tabs into one space. A single tab in the middle of a line also becomes a space, so switch it off for tab separated data.
- Keep indentation leaves the spaces at the start of a line alone while the rest of the line is collapsed. It cannot be combined with Trim each line, which removes indentation by definition.
- Remove empty lines deletes lines that are empty or hold only spaces and tabs. Collapse several empty lines into one keeps a single blank line between paragraphs instead.
- Remove line breaks joins everything into one line, with a space or nothing between the pieces. With Keep paragraph breaks on, lines are joined inside each paragraph and a blank line is left between paragraphs.
- Replace unusual spaces turns non breaking spaces and other special spaces into a normal space. Remove invisible characters deletes zero width spaces, byte order marks and soft hyphens.
- Straighten curly quotes and Replace long dashes swap typographic characters for plain ones. A hyphen is not an exact stand-in for a dash, so read the result.
Stripping HTML: what it does and what it does not
The cleaner reads your text for tags, not as a full web page. It removes tags and comments, drops everything inside script and style elements, and turns line break and block tags such as p, div and li into line breaks so paragraphs do not run together. Entities such as &, and numeric ones like © become the characters they stand for. A less than sign that does not start a tag, as in 5 < 6, is kept.
It does not run, render or fetch anything. It is also not a security sanitizer: do not use it to make untrusted HTML safe to put on a web page. It is for turning markup into plain text you can read.
Quotes, dashes and invisible characters
Curly quotes look better in print, but code, spreadsheet formulas, command lines and many data formats expect straight ones. Pasting “this” into a JSON file or a command breaks it. The quote switch converts curly single and double quotes, including the low and reversed forms used in German and other languages, to plain ones. French style angle quotes are left alone because they are real quotation marks there.
Zero width spaces are the quietest problem. They are invisible, but a search for a word will not find it when one is hiding inside, and two identical looking values will not match. The invisible character switch removes them, along with byte order marks and soft hyphens. It keeps the zero width joiner and non joiner, because emoji sequences and some scripts, such as Persian, need them.