- Home
- Tools
- Text Tools
- Sort & Dedupe Lines
- Remove Duplicate Lines
Remove Duplicate Lines
Paste a list and every line that already appeared is taken out. The first one stays where it was, and the order doesn't change.
- Lines in
- 0
- Lines out
- 0
- Duplicates removed
- 0
What counts as a duplicate
- Before
- apple Apple pear apple pear pear
- After
- apple Apple pear pear
On this page a line is a duplicate when it is exactly the same as an earlier line, character for character. Of the six lines in the example, four are left: Apple has a capital that the first line doesn't have, and the indented pear starts with two spaces.
Three switches loosen that:
- Ignore case treats Apple, apple and APPLE as one line. Accents still count: é is not e.
- Trim spaces around each line removes spaces and tabs at the start and end before comparing. That is usually what you want for a list copied from a spreadsheet or an e-mail.
- Remove empty lines takes blank lines out altogether. Without it, several empty lines are duplicates of each other and one of them stays.
With the first two switched on, the example shrinks to two lines.
The first occurrence is kept, as it was written and where it was. Nothing is sorted unless you pick an order yourself. The counter under the options tells you how many lines went.
Two lines that look identical and both survive differ in something you can't see, such as a no-break space or another kind of apostrophe. The Text Cleaner takes those out.
Sorting, removing duplicates, numbering and adding a prefix can be combined on the Sort & Dedupe Lines page, which also remembers your settings.