I am considering what it takes to implement an email server. Google Cloud basically doesn't allow you to send emails at scale (they block the email ports pretty much), though it sounds like you could receive email. AWS on the other hand allows you to send emails for about $1 per 10k. This sums up some other SMTP services like SendGrid, and the costs involved.
I am aware (vaguely) that there are lots of problems Internet Service Providers (ISPs) want to prevent, like email spam. It sounds like they have IP blacklists, and somehow intercept the emails and can figure out if they are spam by checking their content. Somehow also they get access to abandoned email accounts and check who is emailing there (I have no idea how this works, but if there are some helpful links I'd love to know, though not relevant for the question). Basically, the ISP uses all kinds of techniques to figure out if your email service is spammy, so they can block your IP and shut it off. I don't see why this needs to happen at the ISP level, but that's beside the point.
What I'm wondering about is how to architect an email server so as to not get black listed, and to have it "work" 24/7, for years and years without interruption. Say I want to implement a service like Gmail or SendGrid. I'm wondering what measures you should take to architect an email server. That is, what the best practices are architecture-wise to create a successful email server.
Specifically where I'm at currently is, it seems using Amazon SES is the best option. It is the cheapest by far and doesn't have any bells and whistles. Otherwise you would have to buy your own hardware and build your own cloud if you wanted to get any cheaper or lower level I'd imagine, and buy your own IP addresses. But short of that, using AWS SES sounds like a good option.
They give you the ability to use dedicated IP addresses, and as they state:
most email certification programs require you to have dedicated IPs because they demonstrate your commitment to managing your email reputation.
So email server architecture principle 1, have dedicated IP addresses. But I don't want to do this just yet and then get blacklisted for an unknown reason, which brings me to the crux of the question. How not to get blacklisted. Given this is a service like Gmail or SendGrid, which could be sending millions of marketing emails and millions of personal emails, from millions of different email accounts, every day. I don't see how to tell if I am putting the right things in place for the email server to be top quality and to potentially be "certified" (not sure what email server certification really is or if it's a thing, Google search doesn't reveal anything, but AWS mentions it). That is, what the high level things are that you should put in place to guarantee that all emails will always get delivered (or all emails from all "good" email accounts on your system get delivered). If it's not possible to guarantee this, then I'd like to know why not, and the answer could just be tailored to whatever is closest to a guarantee that we can get.
Basically, the architectural measures to put in place for an email server to consistently deliver email without being blocked.
I am not (for this question) considering anything about scaling the email server or building the email server itself, just the architectural best practices to prevent being blacklisted.
From my understanding so far, some of the initial principles are:
- Have a dedicated IP address. (Not sure if you should just have one, or if you can have 2 or 3, or 100).
- Don't send spam.
That's all I can think of. For (2), this means you have to have good spam filters in place, and other security measures such as verifying that there is a person behind the email account, etc. But for (2) as well, I am unsure how to handle the problem of false negatives. That is, some users might send 100+ individuals a day, maybe even a few mass marketing emails like on those "get rich with adwords" marketing sites with email lists in the 10's of thousands. I would like to know if purely the volume of emails causes a red flag, and how to get around that. Then the content, just want to make sure this is purely based on in-house spam filters, and that the ISP wouldn't block that kind of stuff.
If this is a broad topic, I would like to keep it narrowly focused. I imagine one part of this is to learn more about email spam prevention, which I will do. So this question doesn't need to cover the spam stuff in any detail. To keep it narrowly focused, I'm wondering what architectural measures should be put in place not to be blacklisted. This might include (just making this up):
- Have a fixed number of dedicated IP addresses, less than x number.
- Contact some ISP providers and tell them manually (on the phone even) about your business goals.
- Implement spam filters to prevent spam going out in the first place.
- If you have geographically distributed email servers, perhaps something there as well.
- Programmatically send the abandoned accounts or closed accounts to the ISP for checking.
- Give access of ISP to some other stuff perhaps, by manually creating an API integration and partnership or something.
- Associate phone numbers with the accounts.
I can understand how to implement an email/SMTP server, and send/receive messages at scale. So architecturally that makes sense. What's missing in the picture is the architectural components to prevent being blacklisted at this sort of scale.
To put it succinctly, I'd be interested to know how Gmail and SendGrid avoid getting blacklisted, but that's probably proprietary :)