Remove Duplicate Lines — Clean Up a List in One Step
Paste a list and get it back with the repeats removed — or, if that is what you need, with only the repeats shown and a count of how many times each one appeared. Case and surrounding whitespace can be ignored while comparing, blank lines can be dropped, and the result can be left in its original order or sorted. Nothing is sent anywhere.
This is the small job that turns up constantly: an export of email addresses where half the rows are duplicated by a bad join, a keyword list stitched together from three tools, a set of URLs for a redirect map, a log file where the same error repeats four hundred times. Doing it in a spreadsheet means a formula and a helper column; doing it in a shell means remembering whether sort -u will reorder things you needed in order.
Comparison rules change the answer
Whether two lines are duplicates depends on what you decide counts. [email protected] and [email protected] are the same mailbox but different strings, so for a mailing list you want case ignored. A trailing space from a copy-paste makes two otherwise identical lines distinct, which is why trimming is on by default. For code, config or anything where case is meaningful, turn case-insensitivity off — collapsing userId and userid would be a bug, not a cleanup.
Finding duplicates instead of removing them
Sometimes the repeats are the point. Showing only the duplicated lines, sorted by frequency, turns a list into a quick frequency report — which SKU appears in two feeds, which error dominates a log, which URL was submitted twice. The opposite view, lines that appear exactly once, is how you find the entries present in one source and nowhere else. Both are a click away, with the counts shown alongside.
Order matters more than people expect
The default keeps the first occurrence of each line where it was, which is what you want when the order carries meaning — a playlist, a sequence of steps, a redirect map. Sorting alphabetically uses natural ordering, so item2 comes before item10 rather than after it, which is how a human reads a list and not how a plain string sort behaves. Sorting by frequency puts the most repeated entry at the top.
Related: count what you pasted with the word counter, strip personal data with the text anonymizer, or compare two lists line by line in the diff checker.
How to use Remove Duplicate Lines
- Paste the list — one entry per line. The counter underneath tells you how many lines went in and how many are distinct.
- Choose what to keep: remove the duplicates, show only the duplicates, or show only the lines that appear exactly once.
- Set the comparison rules. Ignore capitalisation for email addresses and names; leave it on for code and identifiers.
- Pick the order — original, A to Z, Z to A, or most frequent first — and turn on counts if you want the number beside each line.
- Copy the result or download it as a .txt file.
Features
- Three modes — remove repeats, list only the repeats with counts, or list only the entries that appear once.
- Case and whitespace control — decide what actually counts as the same line before anything is removed.
- Natural sorting — item2 sorts before item10, the way a person would read it.
- Occurrence counts — turn any list into a frequency report without a spreadsheet.
- Live counts — lines in, distinct, and repeated, updated as you type.
- Runs in your browser — customer lists and exported data are never uploaded.
Frequently Asked Questions
Does it keep the first or the last occurrence?
The first, in its original position. Everything after it is dropped, so the order of the list you pasted is preserved unless you choose to sort the result.
Is the text sent to a server?
No. Everything runs in your browser. That matters because these lists are usually exports — email addresses, customer references, internal URLs — and those should not be pasted into a remote service.
How do I remove duplicates but keep the original order?
That is the default. Leave the order set to "Keep the original order" and only the repeats are removed; the first appearance of each line stays exactly where it was.
Can I see which lines were duplicated?
Yes. Switch to "Show duplicates only" and turn on the counts. Sort by most frequent and you have a frequency report of the list.
Should I ignore capitalisation?
For email addresses, names and tags, yes — the same value written two ways is still the same value. For code, identifiers, case-sensitive IDs or file paths on Linux, no: collapsing them would merge two genuinely different entries.
How large a list can it handle?
Tens of thousands of lines are fine on a normal machine. Very large files — hundreds of thousands of lines — are better handled with sort -u or awk locally, since the whole text is held in the page.
What happens to blank lines?
They are dropped by default, because they are almost always an artefact of the copy rather than data. Turn that option off and empty lines are treated as ordinary values that can be duplicated like any other.