Skip to the content

Remove Duplicate Lines

Paste a list and every line that already appeared is taken out. The first one stays where it was, and the order doesn't change.

Lines in
0
Lines out
0
Duplicates removed
0
Share this tool:TwitterFacebookLinkedInWhatsAppEmail

What counts as a duplicate

Before
apple Apple pear apple pear pear
After
apple Apple pear pear

On this page a line is a duplicate when it is exactly the same as an earlier line, character for character. Of the six lines in the example, four are left: Apple has a capital that the first line doesn't have, and the indented pear starts with two spaces.

Three switches loosen that:

  • Ignore case treats Apple, apple and APPLE as one line. Accents still count: é is not e.
  • Trim spaces around each line removes spaces and tabs at the start and end before comparing. That is usually what you want for a list copied from a spreadsheet or an e-mail.
  • Remove empty lines takes blank lines out altogether. Without it, several empty lines are duplicates of each other and one of them stays.

With the first two switched on, the example shrinks to two lines.

The first occurrence is kept, as it was written and where it was. Nothing is sorted unless you pick an order yourself. The counter under the options tells you how many lines went.

Two lines that look identical and both survive differ in something you can't see, such as a no-break space or another kind of apostrophe. The Text Cleaner takes those out.

Sorting, removing duplicates, numbering and adding a prefix can be combined on the Sort & Dedupe Lines page, which also remembers your settings.

See all 9