I deploying a web scrap server using scrapy. The scenario is a Linux server running mongoDB (or maybe mySQL I'm not sure at all...) and python scripts with scrapy framework.

The main question is what to use to provide data to other servers. I'll try to explain:

I have a delphi app running in another server, the idea is that delphi app ask for some data to the scrap server and scrap server answer it. Some options here:

  1. Scrap server runs scrapy script program sometimes during the day and saves information to database. Delphi app connects directly to database and get the information. I have some idea how to do this, but I don't know if is the best or the most standard option (thinking in future...).

  2. When delphi request information, "call" the server, the serve executes scrapy script and returns CSV. I have no idea how to do this... I read that maybe with PHP??? :S

  3. Maybe with REST or SOAP?

As you can see I'm lost in this topic, any help or comment about the architecture is welcome!!!!

Thanks in advance!

1 Answer 1


You can use a service like Scrapyd

The steps would be as follow:

  1. Deploy spider to scrapyd which saves the items directly to a database
  2. Make the spider crawl through the scrapyd schedule endpoint which will return the following output {"status": "ok", "jobid": "26d1b1a6d6f111e0be5c001e648c57f8"}
  3. Save the jobid
  4. Make calls to the scrapyd listjobs endpoint and check if the jobid is present
  5. If the jobid is not present it's because the spider has finished running
  6. Fetch the data from the database
  • Thanks for your answer. Where is the correct place to ask this?
    – anacelto
    Mar 6, 2017 at 12:04
  • @RafaelAlmeida when referring other sites, it is often helpful to point that cross-posting is frowned upon
    – gnat
    Mar 6, 2017 at 12:53
  • @gnat Thanks, it's however the responsibility of the OP to migrate the question accordingly but glad you contributed, first time that I came across this. Mar 6, 2017 at 12:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.