Remove Duplicate Lines

What counts as the same line

The pass splits the box on \n or \r\n. With Trim each line checked, each row is trimmed before it is tested and the trimmed text is what can be kept. A row that trims to nothing is skipped. The key is the trimmed row, or the lowercased trimmed row when Ignore case when matching duplicates is checked. The first time a key appears, that row is kept. A later row with the same key is skipped. Order is the order of first appearances, not alphabetical order.

With the case box off, the sample keeps apple, banana, and Apple, and the tally is 1 because only the second apple was extra. With the case box on, Apple shares a key with apple, the first spelling apple is the one that stays, and the tally is 2. The kept text is not rewritten to lowercase. Only the key is lowercased. Turning the case box off again restores Apple, because the pass runs from the input on every change.

Trim, case, and the removed tally

Trim each line starts checked. A row of spaces becomes empty and is dropped, and it adds to the tally. Uncheck it when a leading space is part of the value. Then a row of spaces is kept if it is not empty, and apple does not match a row that is apple with spaces around it. The tally's sentence is Removed N duplicate or blank lines. It does not split that N into two figures. An empty box shows Removed 0 duplicate or blank lines. and the result pane says No unique lines found. The copied value in that case is empty.

Ignore case when matching duplicates uses JavaScript lowercasing of the whole row. It is not a full case fold. A character whose lowercase is not what you expect for your language can fail to match, or can match more than you wanted. The box does not apply Unicode normalization, so two rows that look alike but use different code points stay as two rows when trim and case do not erase the difference. The status does not list which code points differed.

When the first copy should stay where it is

Use this page when the list is already one item per row and the first time you saw an item is the row you want to keep. A keyword export, a list of ids, or a log field pasted one value per line fits. Sorting would move the surviving apple next to other a rows and throw away the sequence. This pass does not sort. If the destination later wants A-to-Z order, copy the kept rows and sort that copy.

Wrapped sentences are not joined here. A paragraph that was broken across rows becomes several items, and only exact row matches collapse. Line operations can join those breaks, and it also has a duplicate button that does not trim and does not ignore case. The text sorter orders rows and leaves duplicates in place. The pasted-list guide compares those passes, including when you should join wraps before you deduplicate.

Spaces, case folding, and rows that are not lines

Matching is the whole row. The pass does not look inside a row for a repeated word, and it does not understand a CSV column. A quoted record that contains a break is split into two rows before any key is built. Paste one field per line, or use a CSV tool when the record is the unit. Two rows that differ only by a trailing space collapse when Trim each line is checked, and both survive when it is not.

The sample's second apple is the row the default settings remove. Apple survives those settings because the case box is off. If a row you thought was a duplicate survived, compare spaces, case, and invisible characters before you turn more options on. The tally includes every dropped blank, so a file that was mostly empty lines can show a large N even when few spellings actually repeated. Download output writes the kept rows, one per line, without the tally sentence.

Rules and limitations

Rows split on \n or \r\n. Trim each line starts on and both matches and emits the trimmed text. Empty trimmed rows are dropped. Ignore case when matching duplicates starts off and lowercases the key only, keeping the first spelling.

Order is first-seen order. The tally is input rows minus kept rows and includes blanks. An empty result pane reads No unique lines found. Copy and download are the kept rows. There is no sort on this page.

Common problems

If two rows look the same and both remain, check case, surrounding spaces, and look-alike code points. Case matching is JavaScript lowercase, not every linguistic equivalence.

If a CSV record lost a field, a break inside the record became two rows. This page does not parse quotes or columns.

Frequently asked questions

About this tool

Maintained by Toolbench. For large or deeply nested files, split the input into smaller batches and verify representative results before using them in an import or application. Report reproducible problems through Contact.