To my knowledge multi-threading is just an illusion since CPU schedules time for each of the process. So when scrapping at a particular time, only one content from the website will be scrapped and it will also lead to so much of difficulty since all of the process are urging for the same resource so there might be a problem with "resource allocation" requiring mutex to be implemented.

The ultimate role of mutex is to prevent another thread to access the same resource. So if mutex is implemented, even the perfect implementation of threads would result is in scrapping one info from the website at a particular time.

So how does multi-threading help in scrapping website?

Consider the bot has no GUI(for preventing comments that threading will help in managing GUI and time taking process).

TDLR? Will multithreading speed up web-scrapping process?

  • Write both and then say its an illusion – Ewan Oct 8 '16 at 16:43

First, most of modern processors are multicore and can handle real multithreading at the level of the hardware. Usually, the speed of each core is a little bit slower than if only one core would be used. But the overall processing throughput is much higher.

Second, web scraping consist of requesting resources to a web server. This requires much I/O activity (network protocol exchanges, writing resources harvested to disk, etc...). I/O is much slower than processing, so the CPU would spend much of its time waiting for I/O to be completed. This is why even software based threading that you describe would yield higher throughput: instead of waiting improductively, the CPU passes to the next thread if the I/O is not finished.

Last but not least, web resources are many. If harvesting a page, you might have several pictures, video material, links to other pages. So while this media is served -- even if it would be locked for sequential access only --, there is enough other unlocked data to fetch.

Finally, read access to resources is seldom mutex protected. Mutexes are only required for data race, i.e. concurrent reading and writing. Furthermore, websites are sometimes dispatched or replicated on several servers for higher throughput and load balancing.

So in conclusion: yes, multithreading will definitively improve web scraping.

  • @RobertHarvey fortunately oals corrected my wrong spelling in the meantime (thanks to him/her). I've corrected the wrong link : thanks for pointing it out. – Christophe Oct 8 '16 at 17:51

First, you should have a look what's the bottleneck of your application. Is it CPU? Probably not - or are you working very hard on the data?

My guess is that it's IO (network) bounded. Calls over the network are sloooow, transfer is sloooow. So your single-threaded application does just one thing: waiting for the data to arrive. Got a slow responding server on the other end of the line? Get a cup of coffee.

If you throw several threads at it, they can work on other websites at the same time. That is, up to the point your network connection is saturated. Your system might limit the number of concurrent connections, but it's typically > 1.

To clean up with your opening point: CPU schedules several tasks, but you might have more then one CPU core, incl. hyper threading.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.