is there a good pattern for how to send multiple calls to a web service but without taxing it and ensuring the data is sent back? I don't know enough to correctly describe the problem to even start googling it properly - current google results compare streaming vs. non-streaming wcf answers?

Scenario: I am working on an app (I'm a jr. dev.) that needs to gather information from several sources about a 'customer' and the domains they own

Technical: for one of the sources, I need to send a string array of domains to a web service and this web service returns an entry for each domain name, but this list of domain names will be thousands long - I would like to attempt to divide this list into bite-size chunks (1k domain names each) and then... queue them up to send to that web service, but ensure the web service doesn't skip one

PseudoRequirements: Consumer of web page does not care how long it takes, but would like a list of results up front that does not require pagination to navigate.

Current Theory: Should I take my massive 30k list, break it into 1k chunks, stuff each 1k-sized chunk into a 'request' object, assemble those 'request chunks' into a 'request chunk list' and iterate over that list (sequentially / blocking, so I don't strangle the WS) and for each 'request chunk' get back a 'response chunk' assemble those into a list, and then pass that list back into the front end for viewing? is this a viable method? is there a better way to queue items? Does anyone know off-hand any useful articles for this sort of queuing? are there any 'gotchas' or additional items to consider before I attempt my first pass?

Additional Edits: -I do not have full control to the receiving service, I can not view it's code and the developers that manage it are... less than responsive to email. I do not currently know the stress testing limits of the web service. I emailed the owners of that component but have yet to receive a response - I was going to work up my design while I waited on them.

  • It is very likely that many small requests are going to be far harder on the server than a single large one. Also, 30k doesn't seem remotely "massive" in this day an age.
    – user53141
    Dec 10, 2013 at 20:27
  • You say you don't have enough knowledge to start googling the problem... Try "chatty vs chunky".
    – Mike
    Dec 10, 2013 at 20:46
  • Honestly I'd first make REALLY sure that the human user really wants this feature where "he waits no matter how long in order to get an unwieldy list of data".Wouldn't they much rather prefer to have a paginated presentation? Did they actually see a mock-up of how such a list would actually look, and possibly if they would really,hands-on,tolerate "no matter how long waiting time"?It might just be that they change their opinion on this if you discuss it with them in more detail, and in turn make your job easier (no need to whack your head to see how you can transfer 10 Mb of text via http)... Feb 10, 2014 at 13:25
  • @ShivanDragon, excellent suggestion, I did talk with the user and when he stated that he was just being accommodating due to the fact that I'm a junior developer and he wanted something faster, since this post and talking to him he does want pagination at a later date, do you have any suggestions on how to 'add' pages into the pagination as they come in via AJAX? the only source code I have to copy/learn from in this project is web forms and all the data is gathered, databound, and then presented... I'd love to construct something that adds pages as the results come back
    – Paul42
    Feb 10, 2014 at 18:37
  • Well, I have to first say that the fact that the client made a technical decision for you is not the best thing to have: he decided to simplify your job and in fact made it harder. It's very good that you discussed and clear this out with him. Make sure there aren't any other similar points to clear out. Best way to go about this is to make a Proof Of Concept, even a simple one (just mocks of user interface). A lot of stuff becomes more obvious once you have a visual aid over which to talk. Feb 11, 2014 at 10:53

2 Answers 2


I think your strategy is OK. Other things to consider: Before sending any messages, know how many you plan to send. If you have n messages to send, you might not want to display the full results set until you get all of your responses.

There are other things you could do such as adding correlation IDs or sequence numbers to make tracking all the messages easier, but that would assume you have control over the receiving service, and you don't say if you do.

Other questions: have you actually tested this web service to see what it can reasonably handle in a single request? You might find that it can handle more than you think. Or less than you think. Do you know for certain that a single request with 1000 domain names is optimal?

  • Thanks for your response! -all domains have a domainID, I was planning to match the responses from various sources (Web service and other databases) on this -I don't have control over their web service -I haven't tested their web service, I don't have an optimal size but I'm working on my design while I wait for them to get back to me. Let me know if I should add anything else to make my question more understandable. Thanks again!
    – Paul42
    Dec 10, 2013 at 18:52

Its admirable that you take this into consideration but if the API owners aren't being helpful then your doing your best.

Personally i'd break it down into "normal" traffic chunks, if they are expecting people to throw 1000 at a time to them then do that. Normally this stuff would be listed in the api terms and conditions

With modern internet/networks and webservers you could probably get through those 30,000 records before anyone even had time to complain, last night I did some fairly intensive scraping of 120,000 records from a website in a few hours and simply added a 2 second delay between each request.

The key here is that if they're not getting calls saying people can't access the API then there isn't a problem, since they put it up for you to access and you're not abusing it!

  • after reading, I did just choose an arbitrary amount of 1k records and decided to "ask forgiveness" later if I broke their test environment API, I tried it and it worked like a charm. they were unaware and the hardware was not taxed. I have since increased the limit to 2k and still have no problems. Thank you for your suggestion!
    – Paul42
    Feb 10, 2014 at 18:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.