To use or not to use a backing app for website?

Question

By suggestion in comments, I'll try to rephrase my question to better reflect my problem:

I have to present users (few at first, as many as possible later) with a website. There they will be able to browse various data. The data will be served from a local "cache" db. Users should be always presented with data as fresh as possible.

The source for most of the data will be external services with possible limits, long response times etc. The process of processing data from those sources to the schema of local db is also not straightforward (conversion, calculations, decoupling, recreating relations etc.)

Is putting all that data mining and processing code ok (and doing all that work in the time between users click a link and receive data), or is my second idea - another, constantly running data-miner application - better? Or is there another option?

Old text:

I'm currently developing a website that will present both its own data (user profiles, their submited content) as well as data obtained from various external sources (e.g. via webservice calls).

Varied sources will have varied limits of allowed calls, will respond with varied speed etc.

No, the data user is presented with should be, by default, as fresh as possible with an option to request refresh (unless refreshed very recently).

I started development in ASP.NET (for better or for worse), with some templates and some customization, and quite a lot of coding I ended up with a workable proof of concept. But then I started to think about large-scale operations.

Displaying cached data is quick and easy, I didn't mess that up. Refreshing though is a problem.

a) Requesting data for one user is fast, but it will eat through allowed api calls quickly. b) Requesting data for multiple users (for apis that allow that) saves potentialy great amount of api calls, but takes linearly longer (not 1:1, but still, it takes time to request, receive and parse tens of times more data than single request).

There also comes the problem of exceeding the limits. Usually that would force the app to either cancel refresh, or wait few seconds for new allowance. But I don't really see the user happily waiting for several seconds for the page to load.

I am currently concidering gutting the data-gathering part of the website and moving it into a single managing application, sharing a db with the website. The website would provide info which users need their data to be refreshed, the app would load-balance the external services to use them efficiently (not limited to "when a user opens a website"), and keep the data as fresh as possible - also possibly keeping part of the allowance for forced refreshes.

Now, the main problem I see with this is cost of hosting not just website, but also a background running .NET application. But let's assume that the cost is not that relevant. What I actually fear is that I overcomplicate the solution. I might just put more of the website in Ajax and just wait for the request to complete in the background (when calls become available).

So, does anyone has any simillar experience? I'm especially interested in the scenario where it is all scaled up, that is we can assume that there will always be deficit of available calls.

PS I also worry about keeping website (which can be triggered independently by many users) from trying to refresh the same records twice etc. - I guess I should put some additional safeguards, lock or at least mark the records undergoing refresh etc. - a lot of code just to solve the problem that wouldn't exist with a single-backend....

Answer 1

If you're making calls to a 3rd party service and that service's response is not dependant on the user's request then yes - consider making a data miner gateway between you and the service.

The key points to decide are whether the 'cost' of calling the service is prohibitive, if you are calling it repeatedly to fetch the same data then there's no reason why you need to call it on every request (eg a stock price say, where users get data every 15 minutes, you can simply call it once every 15 minutes to refresh the price). If however, you do need to update it based on the user, then its going to weigh heavily in favour of calling every request and the data miner will not work well.

I have done similar things - usually for data that is received independently of the user requests - eg GPS signals for a vehicle, the data comes in when it likes and it got stored in a dB that the user would read from when the user needed to know where the vehicle was.

Waiting for multiple users to request the same data is a poor choice though - make the call for the first user and cache the results, if another user requests that data within a time limit, respond from the cache instead. Where the cache is not filled with the right data, let the cache make the request to the 3rd party to fill itself while blocking the user request, then the newly-filled cache will be the only interface for all user requests - that kind of architecture makes life easier.