We are currently developing a new email deployment system. We have the system currently configured so that it reads a record from the database and loads the email content and deploys it to the target. Now we want to move this over to multiple threads. That is easily done, except we then hit the email providers returning SMTP codes referring to "Too many connections", or "Deferred connection". The solution to this is to have a thread open up a connection to the email provider and deploy n emails and then disconnect. We have currently configured the application so that it will support these session based email deployments.

The problem is this, the database table has multiple email addresses in and they aren't grouped by email provider because that will show favouritism. We need to be able to retrieve a set number of, i.e. Hotmail, emails (@hotmail.com, @hotmail.co.uk, @live.co.uk) so that we are reducing the number of connections to Hotmail and reducing the risks of getting the "Too many connections" error. We are at the point now where we have gone round and round in circles trying to get a solution, so I thought I'd throw it out there and see if anyone has any ideas?


I would like to stress that this application is not used for spamming purposes.

  • Let me see if I get this straight: you have a table of randomly mixed email addresses, from which you want to select emails in batches by email provider, where the email provider for each batch is consistent across the batch, but is random from one batch to the next? – blueberryfields Oct 20 '11 at 16:19
  • Kind of. Because of the multi-threading it adds an extra complexity to the problem. What the service needs to do is build a "batch" of email addresses, grouped by email provider and send to a thread to deploy too. – Neil Knight Oct 20 '11 at 16:22

In the table that contains the messages that are queued, I'd also include a parsed out provider name (ie, 'hotmail.com', 'yahoo.com') and a flag indicating if the email address is from a 'bulk' provider (the exact meaning of which depends on whatever limits your'e seeing now - if there's a lot of email addresses from a specific provider, mark it as bulk). Have one or more processes/threades work against non-bulk provider emails, and one or more others dedicated to a single bulk provider each.

If you want to pull them back in random order, when you queue the message add a random number. Sort by that first when you pull out addresses for a particular provider. This way they go out in a different order each time.


Divide and conquer is certainly the way to go. Setup an array of queues where each queue is serviced by one smtp connection to a single provider. Load the list of addresses for that provider into the specific queue and let it sned the messages one by one. If for a certain provider/queue the number of addresses is below a certain treshold you can merge them and have an extra connection (to the additional provider) service that specific queue.

  • This is kind of what we have at the moment. The problem is we are talking large volumes of emails, potentially a million. – Neil Knight Oct 20 '11 at 16:34
  • So you are most likele to hit the physical connection limit of a singel machine. To scale this out over multiple machines you can create a service that accepts a range of providers it should email and let this service in turn run the above mentioned process autonomously. Upon completion it can asynchronously notify the coordinator of its completion. – Carlo Kuip Oct 20 '11 at 17:56

In pseudo-code:

dictionary = Dictionary<Host, Queue<EmailMessage>>;
for each email message in original list:

for (i in 1..number of threads):
    in a new thread:
        while (!queue.Empty):
            (host, messages) = dictionary.ReturnAndRemoveARandomItemAtomically();
            send the first X emails in messages to host
            if the message list isn't empty:
                dictionary[host] = messages

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.