Remove Duplicate Lines Online: Complete Guide

Need to remove duplicate lines from a list? Paste it into the free PHL Tools Duplicate Line Remover and get only the unique lines, in the original order. This complete guide explains every option: exact or case-insensitive matching, ignoring spaces, removing empty lines, trimming kept lines, and keeping the first or last copy. It works for keyword lists, emails, URLs, names and log files, shows how many duplicates were removed, and never uploads your list.

Why Remove Duplicate Lines?

Lists grow messy when you collect them from several places. A keyword list merged from three tools, a mailing list from many forms, or a list of URLs for internal links will almost always contain repeats. Counting them by eye is slow and unreliable.

The Duplicate Line Remover compares every line, keeps one copy of each repeated line and reports exactly how many it removed. The order of your list is preserved, so nothing is shuffled.

Step by Step: Clean a List

1. Open the Duplicate Line Remover.
2. Paste your list into the first box, one item per line.
3. The unique lines appear in Unique lines at once.
4. Check the counts under the boxes: lines in, lines kept, duplicates removed and empty lines removed.
5. Press Copy result or Download .txt.

apple
Banana
apple 
banana
cherry

Banana

date

apple
Banana
apple 
banana
cherry

date

With the default options only the exact repeats were removed: the second Banana and the second empty line. apple (with a trailing space) and banana (lowercase) count as different lines, so they stayed. The options below fix that.

The Options Explained

• Ignore upper and lower case: Apple and apple count as the same line. Tick it for keywords, emails and names.
• Ignore spaces at the start and end when comparing: lines that differ only by a space at the beginning or end count as the same. Very common in lists copied from spreadsheets. The kept line is left exactly as written.
• Remove empty lines: deletes blank lines completely and reports them separately from duplicates. Without it, only the first empty line is kept.
• Trim the lines that are kept: also removes spaces at the start and end of the lines that stay. This changes the kept lines, so tick it only when you want clean output.
• Which copy to keep: First occurrence or Last occurrence.

Example With All Cleaning Options

Now tick Ignore upper and lower case, Ignore spaces and Remove empty lines on the same list.

apple
Banana
apple 
banana
cherry

Banana

date

apple
Banana
cherry
date

Four unique items remain from nine lines. apple , banana and the second Banana were recognised as repeats, and both empty lines were removed.

First Occurrence or Last Occurrence?

When a line appears several times, you choose which copy stays. The copy you keep also decides the position of the line in the result.

apple
Banana
apple 
banana
cherry

Banana

date
(all cleaning options on, Last occurrence)

apple
cherry
Banana
date

• First occurrence keeps the earliest copy. Use it when the first entry is the one that matters, such as the oldest record.
• Last occurrence keeps the most recent copy. Use it when later entries are updates.

How Lines Are Compared

Two lines are duplicates when they are identical after the options you chose. Accented characters are compared in a normalised form, so é typed as one character and as e plus an accent count as the same. Empty lines are treated like any other line unless you remove them.

The Buttons

• Copy result: copies the unique lines (Ctrl + Enter, or Cmd + Enter on Mac).
• Select result: highlights the output.
• Download .txt: saves the result as a text file.
• Use result as input: moves the result into the input, handy when you want to run it again with other options.
• Paste, Try an example and Clear all work as in the other PHL Tools.

Real-Life Uses

• SEO: merge keyword lists from several tools, remove repeats, then sort them with the Line Sorter.
• Email and contact lists: remove repeated addresses before importing.
• URL lists: prepare a clean list for internal linking or a sitemap check.
• Students and teachers: combine name lists collected from different forms.

Tips and Limits

• Put one item per line. The tool cannot split a comma-separated list. Use Find and Replace to turn commas into line breaks first.
• Unsure which options to use? Start with all three cleaning options on, then check the result.
• Your list stays in your browser and is not uploaded.

FAQ

Related tool guides

Line SorterText CleanerWord Counter