1

I need to write an application which consumes a forum's content via a REST API and stores threads and posts. The application will act as a bridge layer between the forum and write data to a third application periodically - as close to 'real-time'as possible. The platform is PHP 5.3 / MySQL and probably Symfony with the Zend_Rest client.

My question is what would be an appropriate / performant architecture be for the bridge layer? I imagine I will need to do an initial import of the forum data which will be slow (may take hours). The bridge application will also have a front-end for selectively adding the forum messages to the third application and adding further meta data e.g sentiment (was the message positive or negative in tone). I realise the data import / export could be done with procedural scripts and cron jobs but am wondering if there is a better way.

Many thanks,

2
  • 1
    Not really many choices. A web client is a web client; the HTTP rules are simple and strict. Why don't you build it, measure the performance, and then ask specific questions on Stack Overflow?
    – S.Lott
    Apr 6, 2011 at 10:13
  • I'd like to second S.Lott's sentiment.
    – dirk
    Apr 11, 2011 at 3:29

1 Answer 1

1

There's a variety of challenges when it comes to synchronizing API's like this. Although there isn't a silver bullet that I can offer that will make everything work; I can offer a few things to keep in mind.

Take advantage of any type of temporal meta-data available, Last-Updated and similar are your best friend when dealing with the time-oriented nature of forums. Seeing as it is REST, the bridge layer will likely need to run as a cron job. Since you won't be able to register any type of callback to receive new forum posts as they come in, everything will have to be after the fact and rely solely on temporal meta-data. This type of stuff is tricky, and sadly I can't offer a magic solution.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.