Exact match, ignore case, ignore spaces
By default two lines match only when they are identical, character for character. 'Apple', 'apple' and 'apple ' with a trailing space are three different lines. Two switches loosen that:
- Case sensitive off makes Apple and apple the same line. The copy that stays is shown as it was typed.
- Trim spaces on ignores spaces, tabs and other blank characters, such as non breaking spaces, at the start and end of each line, and removes them from the result too. Trailing spaces are the most common invisible reason for duplicates that will not go away.
For email addresses, tags and keyword lists, turn case sensitivity off and trimming on. For code, file names and anything where capitals carry meaning, leave them as they are. Matching is on whole lines only, so 'apple pie' and 'apple' are different, and misspellings are not detected. Lines are compared after Unicode normalization, so an accented letter typed as one character matches the same letter typed as a base letter plus a combining accent.
Keeping the first or the last copy
When a line appears several times, one copy stays and the rest go. Take the list a, b, a, c, b. Keeping the first copy gives a, b, c. Keeping the last copy gives a, c, b, because the last a comes before the last c and the last b.
The first copy is the usual choice, since it keeps items in the order you met them. Keep the last when later entries replace earlier ones: a log of settings changes, or a list that grows downward where the newest line is the one that counts.
Finding duplicates instead of removing them
Two other views are for finding rather than removing. Show only repeated lines lists each line that appears more than once, one copy each. Show only lines that appear once lists the lines that never repeat. The Most repeated lines panel adds the counts.
That makes a quick way to compare two lists. Paste list A and then list B into the box. If neither list has repeats inside itself, Show only repeated lines gives the items that are in both lists, and Show only lines that appear once gives the items that are in only one of them. If either list has internal repeats, remove them first.
Command line equivalents, and limits
On a command line, sort file.txt | uniq removes repeated lines, but uniq only notices repeats that sit next to each other, so the list must be sorted first, and the order is lost. sort -u file.txt does both in one go. To keep the original order and the first copy of each line, awk '!seen[$0]++' file.txt is the usual answer.
This page runs in your browser. Lists of tens of thousands of lines are fine, and bigger ones work but typing in a very large text box can feel slow. For huge files, a command line tool is a better fit. When you also want the result in order, follow up with sort lines.