Finding duplicate email addresses. How heavily should I normalize them?

Question

I have a big list of email addresses and I want to find out which of those are duplicates.

How to define "duplicate" is exactly what I'm posting about here.

I know from experience that with Gmail, it's possible to remove all the periods and the mail will go to the same destination. However, from what I've gleaned this is not uniformly true, and some email providers do consider periods part of the unique identifier.

What's your recommendation for how to handle this? I want to err on the side of caution (definitely don't mark an email address as duplicate if it's not). But I don't want to be held back by a tiny edge case either.

I'm using Ruby, FWIW, but could also comfortable use Javascript, Shell, or Python utils. I already know of a Ruby gem to smartly normalize email addresses (different hosts are treated differently), but I don't want to count on this 100% to do the work for me. So I'm trying to figure out, in abstract terms, what I want to do before I worry about implementation.

Answer 1

Based on additional information in the comments I would advise that you don't attempt to normalize at all except for domains where you know you can do so safely.

If gmail document that they violate/extend standards in some way by ignoring full-stops or allowing 'plus-addressing' then cover them directly but don't apply that on a blanket basis.

You may be able to get other hints such as spotting domains which use services you can normalise by the IP in the MX record for the domain.

You may be able to obtain information about a specific domain through statistical analysis of your dataset.

Add others on a case-by-case basis where you can clearly demonstrate that a specific optimization applies.

Note though that you need to be careful how you use 'sanitised' data as I for one give out plus-addresses and treat mail that doesn't go to the right place as spam.