Duplicate Line Finder
Working with large lists - whether exported from a spreadsheet, copied from a database, or assembled by hand - almost always means dealing with duplicate entries. Duplicates creep in silently: a name entered twice, a product code pasted from two sources, an email address submitted more than once in a form. Left unchecked, they corrupt counts, skew analyses, and cause all sorts of downstream headaches. This tool gives you a fast, no-fuss way to get your list clean before it causes problems.
Paste any list with one item per line and the result updates as you type. Remove duplicates gives you a clean list, Show duplicates lists only the repeated lines, Lines that appear once keeps only the lines that are never repeated, and Count every line shows how often each line occurs. Options let you ignore differences in capitals and spaces, so “London”, “london ” and “LONDON” count as the same entry, and a report shows each duplicate with the line numbers where it appears.
INYour list
OUT
Duplicate report
About this tool
How it works
Your text is split into lines, and each line is compared with the others after the options you chose: with Ignore case, capitals don't matter; with Trim spaces, spaces and tabs at either end are ignored; with Collapse inner spaces, runs of spaces, tabs and non-breaking spaces inside a line count as one space. The output keeps your original text, tidied only by the spacing options you turned on.
Windows (CRLF) and Mac line endings are handled, so a list pasted from Excel or Notepad compares correctly, and a final line break doesn't count as an extra blank line.
Four ways to see your list
- Remove duplicates: every line once, in its original order.
- Show duplicates: only the lines that occur more than once.
- Lines that appear once: only the lines that are never repeated.
- Count every line: each line with how often it appears, separated by a tab so it pastes into two spreadsheet columns.
Choose whether to keep the first or the last copy of a repeated line, and sort the result A–Z, Z–A or by how often lines repeat.
The duplicate report
Below the result, the report lists every repeated line with how many times it appears and the line numbers where it was found. When lines only match because of the options, for example Apple and apple, it also shows the different spellings, so you can see what will be merged.
Common uses
- Cleaning email or mailing lists before importing them into a CRM or newsletter tool
- Removing repeated SKUs, order IDs or reference codes from exports
- Finding repeated survey or form responses
- Merging lists of tags, keywords or URLs from several sources
- Checking data before importing it into a database column that must be unique
Private and fast
Everything happens in your browser, so your list is never sent to a server or stored. Only your option settings are remembered for next time. Lists of hundreds of thousands of lines are processed in a moment, and you can open a text or CSV file of up to 25 MB.
Frequently Asked Questions
What does this duplicate line finder do?
This tool finds and removes duplicate lines from any line-separated list. Paste your list into the input box and the result updates as you type. Remove duplicates returns your list with each entry once. Show duplicates lists only the entries that appear more than once, Lines that appear once keeps only the entries that are never repeated, and Count every line shows how often each entry occurs. A duplicate report shows each repeated line with the line numbers where it was found.
How does the duplicate detection work?
The tool splits your input into lines, so each line is one entry. Each line is compared after the options you chose (ignore case, trim spaces, collapse inner spaces), and the result keeps the first or last copy of each entry, as you choose. The figures under the result show how many lines were checked, how many unique values and duplicated values there are, and how many extra copies were removed.
What is the difference between case-sensitive and case-insensitive matching?
By default the tool is case-sensitive, meaning Apple and apple are treated as two distinct entries.
Turning on Ignore case makes them match. The comparison works for every language, but the output keeps the original capitals of the copy you chose to keep.
This is useful when working with data from multiple sources where capitalisation may be inconsistent, such as city names, product categories, or user-submitted tags.
Is my data safe to use with this tool?
Yes. Everything happens entirely in your browser. No data is sent to any server, stored anywhere, or logged in any way. You can disconnect from the internet and the tool will continue to work perfectly. Your list isn't saved; only your option settings are remembered in your browser. This makes it safe to use with sensitive lists such as customer data, internal codes, or personal information.
What are the most common uses for a duplicate line finder?
Typical use cases include:
- Cleaning email or mailing lists before importing into a CRM or newsletter tool
- Deduplicating product SKUs, order IDs, or reference codes from exports
- Auditing survey or form responses for repeated submissions
- Comparing lists of tags, categories, or labels assembled from multiple sources
- Checking whether a list of usernames, URLs, or file names contains repeats
- Preparing data for import into a database that enforces uniqueness constraints
How do I copy the results?
Use the Copy button above the result to copy the whole result to your clipboard, or Download to save it as a text file. Use makes the result your new list, so you can run another pass; press Ctrl+Z to undo it. You can also select and copy part of the result directly.
Are blank lines treated as duplicates?
Not by default. Ignore blank lines is switched on, so empty lines (and lines containing only spaces) are left out of the result and the counts. Switch it off to treat blank lines as entries, in which case one blank line is kept when you remove duplicates. A line break at the very end of your list never counts as a blank line.
Does the tool account for leading or trailing spaces?
By default, no: Trim spaces is switched on, so apple (with a leading space) and apple count as the same entry, and the result is trimmed. Switch it off if spaces matter in your data.
Collapse inner spaces also treats several spaces or tabs inside a line as one, so New York matches New York.
Is there a limit to how many lines I can process?
There's no fixed limit. Lists of hundreds of thousands of lines are processed in a moment on a modern device, and you can open text or CSV files of up to 25 MB.
Why do my results look wrong after pasting from Word or Excel?
Rich-text sources such as Word or Google Docs can add invisible characters, such as non-breaking spaces, that look like normal spaces but don't match. Turn on Collapse inner spaces and Trim spaces to treat all of them as ordinary spaces. Windows line endings from Excel or Notepad are handled automatically. If lines still don't match, the duplicate report's spellings column helps you see what differs. Smart quotes (’ instead of ') are still different characters.