2

I have a big list of email addresses and I want to find out which of those are duplicates.

How to define "duplicate" is exactly what I'm posting about here.

I know from experience that with Gmail, it's possible to remove all the periods and the mail will go to the same destination. However, from what I've gleaned this is not uniformly true, and some email providers do consider periods part of the unique identifier.

What's your recommendation for how to handle this? I want to err on the side of caution (definitely don't mark an email address as duplicate if it's not). But I don't want to be held back by a tiny edge case either.

I'm using Ruby, FWIW, but could also comfortable use Javascript, Shell, or Python utils. I already know of a Ruby gem to smartly normalize email addresses (different hosts are treated differently), but I don't want to count on this 100% to do the work for me. So I'm trying to figure out, in abstract terms, what I want to do before I worry about implementation.

  • How many addresses are you normalising? And what makes you believe that you are not heading into premature optimization territory? – James Snell Dec 28 '16 at 1:46
  • @JamesSnell potentially millions, and yeah, this is a business requirement - a work thing. To blindly normalize all the email addresses would be premature optimization, but that's why I'm asking here first! – max pleaner Dec 28 '16 at 1:48
  • How do you define "duplicate"? Are "myfirstname@myfullname.example" and "myrole@mycompany.example" duplicates? They go to the same mailbox! Are "myname@gmail.com" and "myname+stackoverflow@gmail.com" duplicates? I wrote a script that delivers them to different mailboxes! What about "myfirstname@myfullname.example" and "admin@mycompany.example"? They go to the same mailbox on weekends but different mailboxes during weekdays. – Jörg W Mittag Dec 28 '16 at 14:51
  • In addition to the dot, be aware that Gmail will ignore anything after a plus sign in the part of an email address before @ Thus john.doe@gmail.com is equivalent to john.doe+is.da.greatest@gmail.com – Mawg Nov 22 '18 at 15:30
6

Based on additional information in the comments I would advise that you don't attempt to normalize at all except for domains where you know you can do so safely.

If gmail document that they violate/extend standards in some way by ignoring full-stops or allowing 'plus-addressing' then cover them directly but don't apply that on a blanket basis.

You may be able to get other hints such as spotting domains which use services you can normalise by the IP in the MX record for the domain.

You may be able to obtain information about a specific domain through statistical analysis of your dataset.

Add others on a case-by-case basis where you can clearly demonstrate that a specific optimization applies.

Note though that you need to be careful how you use 'sanitised' data as I for one give out plus-addresses and treat mail that doesn't go to the right place as spam.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.