Remove Duplicate Lines
Remove duplicate lines, optionally ignoring case and whitespace
What is Remove Duplicate Lines?
A duplicate line remover keeps the first occurrence of every line and drops each later repeat, with options to ignore case, trim leading and trailing whitespace, and decide whether blank lines survive. It also reports how many lines were removed so you can sanity-check the result. Processing happens locally in your browser.
How to use Remove Duplicate Lines
- 1Paste your content with one item per line.
- 2Turn on "ignore case" and "trim whitespace" if your data needs them.
- 3Check the counter at the bottom to confirm the number of removed lines matches what you expected.
How do I do this in code?
Use the tool above for one-off work; for anything you repeat, move it into a script or your project.
# Deduplicate while preserving the original order awk '!seen[$0]++' input.txt # Sort then deduplicate (this changes the order) sort -u input.txt
Frequently asked questions
Why are two lines that look identical not being deduplicated?+
Almost always because of invisible characters: a trailing space, a mix of tabs and spaces, or a zero-width character carried over from a document. Enabling "trim whitespace" handles the first two; zero-width characters have to be stripped with find and replace first.
Does deduplication reorder my lines?+
No. Each value stays at the position where it first appeared, so the output order matches the input. That is different from sort -u on the command line, which sorts the data before removing duplicates.
Related tools
All toolsText Diff Checker
Compare two blocks of text line by line, optionally ignoring whitespace
JSON Diff
Compare two JSON documents structurally, by key path
Regex Tester
Test regex matches live and inspect every capture group
Case Converter
Convert identifiers between camelCase, snake_case, kebab-case, and more
Word Counter
Count words, characters, lines, and estimated reading time
AI Token Counter
Count AI tokens and estimate what a call will cost