I am developing a new project where I will be constantly checking a webpage for data and adding this data to a queue for processing. This data will then be removed from the queue and added to a list if it hasn't already been added. This list will then be stored in the database.

I was wondering about how to use threads as I am relatively new to programming? I think it would be ideal if I had a thread constantly checking the webpage for data, a thread processing the queue and a thread for processing the list.

Any suggestions on whether this method is the most effective or not and how to achieve this would be much appreciated.

  • The webpage is a local page so it won't require leaving the local network.
    – user99903
    Aug 18, 2013 at 14:25
  • 1
    Something is updating the web page. Can't it also put the information of the update in a message queue too? Does anything bad happen if you miss an update of the web page?
    – user40980
    Aug 18, 2013 at 15:56

2 Answers 2


Unless the data on the website changes very frequently (multiple times per second), there is no reason to use multiple threads for this kind of application.

You can just use a big loop that, in order

  • retrieves data from the site
  • updates the list, if needed
  • updates the database with the new list, if needed.

Multiple threads and a queue for communicating between them only adds overhead to the program, for which there seems to be no reason judging by your description of what the program needs to do.

  • The data will change approximately once a minute, sometimes the data may not change for five minutes or longer. I already have a loop that does the aforementioned I just wanted to ensure that there was no more efficient way to do this! Thank you for your response :))
    – user99903
    Aug 18, 2013 at 14:33

To build on MichaelT's comment to your post, if the website is a local website that you have complete control of then you might be better off having it publish a notification of change that your app subscribes to.

Queues/topics will work for this, and it will give you the ability to scale up if required.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.