filemixer

FileMixer / Guides / Remove duplicates

How to remove duplicates from a CSV file

Removing duplicates takes one click. Choosing which rows count as duplicates is the part that needs thought.

1. Decide what a duplicate is

Two rows can be duplicates in different senses:

  • Exact copies. Every cell is the same, usually because the same export was pasted in twice. Compare all columns.
  • The same thing, recorded twice. The same customer signed up twice with a different phone number. Compare only the column that identifies them, such as the email.

Comparing fewer columns removes more rows. If you dedupe a mailing list on email alone, two people sharing an inbox become one row. That's usually what you want for a mailing list, and not what you want for an order history.

2. Decide which copy to keep

Exports are usually sorted oldest first, so the last copy is the newest: the latest address, the current phone number. Keep the first copy when the original entry matters, such as the first signup date.

3. Remove them

  1. Open the remove duplicates tool and add your file.
  2. Uncheck every column except the ones that define a duplicate.
  3. Choose to keep the first or last copy.
  4. Leave Ignore capitals and extra spaces on, so “Ann@Example.com ” and “ann@example.com” count as the same.
  5. Read the count above the preview and download.

Open the tool

4. Check what you removed

Before you trust the result, look at what went. Choose Only the duplicates that were removed and scan the preview. If you see rows that clearly aren't the same thing, you compared too few columns.

In Excel instead

Excel has Data › Remove Duplicates, which works the same way: check the columns to compare. Two differences: it always keeps the first copy, and it doesn't ignore trailing spaces, so “ann@example.com ” survives. Run TRIM on the column first. Opening a CSV in Excel can also change your data (see leading zeros), so the browser tool is safer for CSVs that go back into another system.