I deploying a web scrap server using scrapy. The scenario is a Linux server running mongoDB (or maybe mySQL I'm not sure at all...) and python scripts with scrapy framework.
The main question is what to use to provide data to other servers. I'll try to explain:
I have a delphi app running in another server, the idea is that delphi app ask for some data to the scrap server and scrap server answer it. Some options here:
Scrap server runs scrapy script program sometimes during the day and saves information to database. Delphi app connects directly to database and get the information. I have some idea how to do this, but I don't know if is the best or the most standard option (thinking in future...).
When delphi request information, "call" the server, the serve executes scrapy script and returns CSV. I have no idea how to do this... I read that maybe with PHP??? :S
Maybe with REST or SOAP?
As you can see I'm lost in this topic, any help or comment about the architecture is welcome!!!!
Thanks in advance!