Duplicate Line Remover

Paste a list and get only the unique lines, in the original order. See exactly how many duplicates were removed.

100% Client-Side Processing

Your list (one item per line)

Unique lines

What to output

Ignore upper and lower caseApple and apple count as the same line Ignore spaces at the start and end when comparingThe kept line is left exactly as written Remove empty lines Trim the lines that are keptChanges the kept lines

Which copy to keep First occurrence Last occurrence

0Lines in

0Lines kept

0Duplicates removed

0Empty lines removed

Ctrl + Enter (Cmd + Enter on Mac) copies the result. Nothing you type is sent anywhere.

How to use

Paste your list into the left box, one item per line.
Choose how lines are compared: exact match by default, or ignore case and surrounding spaces.
Pick whether to keep the first or the last copy of each repeated line.
Copy the unique lines from the right box, or download them as a text file.

Features

Exact duplicate removal that keeps the original order of the lines.
Keep the first or the last occurrence.
Optional case-insensitive comparison and optional trimming before comparison.
Optional removal of empty lines, reported separately from duplicates.
Counts of lines in, lines kept, duplicates and empty lines removed.
The retained lines are never altered unless you tick Trim the lines that are kept.

How lines are compared

Two lines are duplicates when they are identical after the options you chose. Accented characters are compared in normalised Unicode form, so é typed as one character and as e plus an accent count as the same line. Empty lines are treated like any other line unless you remove them, so by default only the first empty line is kept.

Examples

Input

apple
Banana
apple
cherry
banana

Default

apple
Banana
cherry
banana

Input

apple
Banana
apple
cherry
banana

Ignore case

apple
Banana
cherry

FAQs

Does it change the order of my lines?

No. Lines keep their original order. With "First occurrence" each repeated line stays where it first appeared. With "Last occurrence" it stays where it last appeared.

Are lines with different capitals duplicates?

Not by default. Tick "Ignore upper and lower case" to treat Apple and apple as the same line.

What happens to empty lines?

By default empty lines are treated like any other line, so only one is kept. Tick "Remove empty lines" to delete them all.

Can I see which lines were removed?

The page shows how many lines were removed. To inspect them, compare the input and output side by side, or sort the input first with the Line Sorter so repeats sit next to each other.

How large a list can I use?

Hundreds of thousands of lines work in a modern browser. Input above two million characters is rejected to protect your browser.

Is my list uploaded?

No. The page works in your browser and sends nothing to a server.

About this tool

Duplicate Line Remover cleans up lists of emails, keywords, URLs, names, log lines and anything else that is one item per line.

It performs exact comparison with the options you choose. It does not guess which lines are similar.

Keyboard

Ctrl + Enter: Copy the result (Cmd + Enter on Mac).
Esc: Closes an open information panel or menu. It never clears your text.
Tab: Moves between boxes and buttons.

Related tools

Line SorterSort lines A to Z, Z to A, by length or by number, with natural sorting.
Text CleanerRemove extra spaces and blank lines, fix line endings and strip unwanted characters.
Find and ReplaceFind and replace text with match counts, whole-word and regular expression options.
More in Data & List Tools

Privacy

Your text is processed by JavaScript in your browser tab. It is not sent to us or anyone else, and it is not saved. It is still visible to you and to any browser extensions you use, so clear it on shared computers. Read the policy.