How can I deal with a slow API in PHP?

Question

I'm writing a public web app to get stock data from a magento store. I've accessed the data, but it turns out that i have to query each product individually for stock data. With thousands of items this is a resource hog and takes lots of time. This cannot be avoided. It's just how their API works and I have to do it this way.

How can I code something in php that runs in the background, harvesting the api data by nibbling it in a calmer more friendly manner, and not hammer their shops with SOAP api calls?

Am I right in thinking that I need to use a background process like this: $pid = shell_exec(sprintf('%s > /dev/null 2>&1 & echo $!', $command));

...and then query that $pid? Store them in a database? Are there any examples of background process managers?

Answer 1

Taking Mike and Thorsten's comment - if this data is not time sensitive - you can create a cron job that will get these data and put it in your own SQL table and then have your App pull from it. Majority of good API calls typically have a pull all API request handler with a delimiter so that you can retrieve bulk data as they don't want you to hammer them because depending on their firewall configuration it might end up being a brute force spam program that wants to poll if you are polling too often. So in conclusion - it might not be you but them.