I'm looking for hints and suggestions on a design for a multi-threaded URL fetcher in java. Specific requirements are:

  • To fetch each one of around 1,000 URLs periodically
  • The interval between each fetch will be URL specific
  • Intervals are likely to be 2 mins to 1 hour

I'm imagining I will need a bunch of fetchers each running in their own thread that get pushed the next URL to fetch when in a "ready" state.

I will need to handle errors, e.g quit querying a specific URL if it repeatedly times out or 404s.

Any ideas much appreciated.


3 Answers 3


1000 threads (some of them active once in 2 hours) is a big no-no. So is starting a new thread for each job which may finish few seconds later. Make one "scheduler" thread that selects URLs for retrieval, and a number of worker threads that report their state to the scheduler. Scheduler sequentially:

  • performs worker thread pool management: -- if no threads are free, spawns some new ones. -- if more than X threads (say, 3) are idle, ends extra threads.
  • selects new URL to retrieve at the moment (or skip the step),
  • finds the first free thread, assigns it the job,
  • collects results from threads that finished (if any)

Then sleep and repeat the loop. Essentially, you have a semi-realtime parent thread that does all "fast" jobs and worker threads that have busy-wait states.

Of course the URL distribution can be done through Observer pattern, modified to "consume" the message if a "client" accepts it (hand out URL to retrieve). The list of threads can be a linked list to be traversed recursively.

  • Yes absolutely, was not thinking of a 1,000 threads! Thanks for the suggestions, very helpful.
    – Richard H
    Jul 5, 2011 at 12:22
  • 1
    And before you start to reinvent the wheel, make sure you read into java.util.concurrent.ScheduledThreadPoolExecutor and related classes. They can handle most of the work you need to create a threadpool.
    – Dorus
    Jul 5, 2011 at 22:10

Check out ScheduledExecutorService. It should have everything you need.

If you want more control, consider using a DelayQueue and program the logic around it yourself.


It would appear that you want a thread-safe priority queue. The priority will be based on the time when each URL should be fetched. When you have a task to fetch a URL, you'll put it into the queue. You'll have a thread pool to service the tasks in the queue. You'll probably want to adjust the exact number on some statistics about 1) how long each URL takes to fetch on average, and 2) how many URLs you have to fetch in a given time.

For example, if you average 3 seconds to fetch the data for one URL, and one URL to fetch every second, then you obviously need at least three threads to come reasonably close to keeping up. Less obviously, you'll probably want to compute the variance (or standard deviation) of the traffic, to get an idea of how many threads you'll need to handle bursts. Alternatively, you can adjust the number dynamically, based on demand over the next few seconds.

You'll probably want to include a repetition interval in the definition of a task, along with number of (consecutive?) times that task has failed. After N repetitions, you simply won't put that task back int the run queue (or, alternatively, you might schedule it to be tried at longer intervals, and possibly only when the system is otherwise idle).

Offhand, it sounds to me like you're probably over-estimating the number of threads involved unless most of those URLs end up a lot closer to the 2 minute interval than the 1 hour interval. If we assume roughly random distribution, we get something like one URL to fetch every ~1.8 seconds. At that rate, one thread may easily be adequate.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.