4

I am using curl_multi_exec to process over 100K requests. I do 100 requests at a time because curl_multi_exec can only handle 100 requests at a time to eventually get to 100K requests. We have added multiple servers to this system to spread the load [we are using load balancing]. What is the best way to have curl handle 100K requests and make use of these additional servers? What is the downside (other than time) of handling that many requests on one server? How can I use the additional servers to help handle those requests?

I was thinking about having each server handle a batch of requests (like one server handles 500 requests, another 500, another 500, etc..

To elaborate- basically, we are using curl to send out over 100K requests to third party servers. The problem with only using 1 server is that there is a memory limit in the number of requests 1 server can handle. So we decided to add additional servers, but we are not sure how to design this system to use curl to handle that many requests..

The third party server is an API like Facebook; they are aware that we will be making that many requests to their servers.

For load balancing, we use Rackspace cloud server, basically the load balancer directs incoming requests to separate servers.

Improve this question
  • Could you describe your problem in more detail? Can you aggregate requests and gather data from multiple proxy servers each making 100 requests at a time? Do you really need that much requests on a single machine? – vissi Dec 27 '11 at 22:16
  • What do you use for load balancing? – vissi Dec 27 '11 at 22:17
  • 2
    Are you hitting one third party server with a whole lot of requests? That level of activity might be viewed as hostile. – David Thornley Dec 27 '11 at 22:23
  • 1
    Is this incoming or outgoing? Why do you need a load balancer if you are sending outgoing curl requests? – Jordan Dec 28 '11 at 5:27
  • it is outgoing. I need a load balancer bc the third party server responds back. I am trying to figure out what the best way to design the system considering I need to send 100K outgoing requests, and that takes time without crashing the server.. – Pota Onasys Dec 28 '11 at 18:10
3

The way you are doing load balancing seems a bit off to me. Rackspace cloud load balancing sits in front of your servers and balances the incoming requests between them. This is great for balancing incoming requests.

However, you are doing outgoing requests. This should be a daemon running on your server(s). And if it is, then your frontend load balancing does not have any effect at all. What you need to do is split up the tasks you are doing and have workers process them in parallel.

The way you usually do this is by using some kind of message or job queue. Gearman is a possibility. You could also use a message queue like rabbitmq. I personally really like zeromq.

Unlike other queueing systems, zeromq is not a daemon. It's not a server that acts as a queue. It's a library that allows you to build queues and concurrency. I highly suggest you watch this presentation.

What this allows you to do is create any amount of workers that do work. In your case they will send API requests. And then you can create a sink that collects the results of those calls. So each worker has its own process and does curl multi on 100 requests. And if you spawn 100 or 1000 worker processes (can be distributed between machines, connected over TCP), they will be able to do do the work in parallel.

I think it fits your use case pretty well: ZeroMQ.

Improve this answer
0

Given the background provided so far, it sounds like Gearman could be useful: http://gearman.org/index.php

It's a great tool that is very easy to use within PHP, and it's built specifically to help

...farm out work to other machines or processes...

There are some great use cases that are documented within the site that can help you get started.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.