3

We have a crawling engine that uses curl and caters to about 400,000 ppl / month. Although curl supports concurrent downloads, it does not support bandwidth limiting ( only in php 5.4.0 ) which is why my boss wants me to move to aria2. He also says Aria2 is faster ( which seems to be true to me as well )

Aria2 would require a large amounts of changes to the system. Right now we have crawling system which does not crawl in a conconrrent manner.

Would moving to aria2 be a good decision?

2 Answers 2

3

After running quiet a few test we found aria and curl/php/apache are running at almost the same speed. We decided to stick to curl as it has a php library libcurl. We are not sure about bandwidth limiting but are working on it. Changing our current code to crawl multiple pages simultaneously was not such a big hassle as of curl. Although there have been some things we need to work around e.g. https://stackoverflow.com/questions/8878546/curl-multi-crawling-issues

2
  1. Bandwidth limiting is supported by PHP Ext cURL. The settings you are looking for is: curl_setopt($ch, CURLOPT_MAX_RECV_SPEED_LARGE, 10240);
  2. aria2c is faster than cURL on mirrored downloads, because it allows fetching parts from multiple sources in parallel and combines them. this is not supported by cURL, afaik. rebuilding this feature on PHP side is possible, but the handling is slower than with aria2c.

Not the answer you're looking for? Browse other questions tagged or ask your own question.