How can I optimally consume and re-syndicate a REST web service

Question

I need to write an application which consumes a forum's content via a REST API and stores threads and posts. The application will act as a bridge layer between the forum and write data to a third application periodically - as close to 'real-time'as possible. The platform is PHP 5.3 / MySQL and probably Symfony with the Zend_Rest client.

My question is what would be an appropriate / performant architecture be for the bridge layer? I imagine I will need to do an initial import of the forum data which will be slow (may take hours). The bridge application will also have a front-end for selectively adding the forum messages to the third application and adding further meta data e.g sentiment (was the message positive or negative in tone). I realise the data import / export could be done with procedural scripts and cron jobs but am wondering if there is a better way.

Many thanks,

Answer 1

There's a variety of challenges when it comes to synchronizing API's like this. Although there isn't a silver bullet that I can offer that will make everything work; I can offer a few things to keep in mind.

Take advantage of any type of temporal meta-data available, Last-Updated and similar are your best friend when dealing with the time-oriented nature of forums. Seeing as it is REST, the bridge layer will likely need to run as a cron job. Since you won't be able to register any type of callback to receive new forum posts as they come in, everything will have to be after the fact and rely solely on temporal meta-data. This type of stuff is tricky, and sadly I can't offer a magic solution.