Get data from scrap server

Question

I deploying a web scrap server using scrapy. The scenario is a Linux server running mongoDB (or maybe mySQL I'm not sure at all...) and python scripts with scrapy framework.

The main question is what to use to provide data to other servers. I'll try to explain:

I have a delphi app running in another server, the idea is that delphi app ask for some data to the scrap server and scrap server answer it. Some options here:

1. Scrap server runs scrapy script program sometimes during the day and saves information to database. Delphi app connects directly to database and get the information. I have some idea how to do this, but I don't know if is the best or the most standard option (thinking in future...).

2. When delphi request information, "call" the server, the serve executes scrapy script and returns CSV. I have no idea how to do this... I read that maybe with PHP??? :S

3. Maybe with REST or SOAP?

As you can see I'm lost in this topic, any help or comment about the architecture is welcome!!!!

Thanks in advance!

Answer 1

You can use a service like Scrapyd

The steps would be as follow:

1. Deploy spider to scrapyd which saves the items directly to a database
2. Make the spider crawl through the scrapyd schedule endpoint which will return the following output {"status": "ok", "jobid": "26d1b1a6d6f111e0be5c001e648c57f8"}
3. Save the jobid
4. Make calls to the scrapyd listjobs endpoint and check if the jobid is present
5. If the jobid is not present it's because the spider has finished running
6. Fetch the data from the database