By suggestion in comments, I'll try to rephrase my question to better reflect my problem:
I have to present users (few at first, as many as possible later) with a website. There they will be able to browse various data. The data will be served from a local "cache" db. Users should be always presented with data as fresh as possible.
The source for most of the data will be external services with possible limits, long response times etc. The process of processing data from those sources to the schema of local db is also not straightforward (conversion, calculations, decoupling, recreating relations etc.)
Is putting all that data mining and processing code ok (and doing all that work in the time between users click a link and receive data), or is my second idea - another, constantly running data-miner application - better? Or is there another option?
I'm currently developing a website that will present both its own data (user profiles, their submited content) as well as data obtained from various external sources (e.g. via webservice calls).
Varied sources will have varied limits of allowed calls, will respond with varied speed etc.
No, the data user is presented with should be, by default, as fresh as possible with an option to request refresh (unless refreshed very recently).
I started development in ASP.NET (for better or for worse), with some templates and some customization, and quite a lot of coding I ended up with a workable proof of concept. But then I started to think about large-scale operations.
Displaying cached data is quick and easy, I didn't mess that up. Refreshing though is a problem.
a) Requesting data for one user is fast, but it will eat through allowed api calls quickly. b) Requesting data for multiple users (for apis that allow that) saves potentialy great amount of api calls, but takes linearly longer (not 1:1, but still, it takes time to request, receive and parse tens of times more data than single request).
There also comes the problem of exceeding the limits. Usually that would force the app to either cancel refresh, or wait few seconds for new allowance. But I don't really see the user happily waiting for several seconds for the page to load.
I am currently concidering gutting the data-gathering part of the website and moving it into a single managing application, sharing a db with the website. The website would provide info which users need their data to be refreshed, the app would load-balance the external services to use them efficiently (not limited to "when a user opens a website"), and keep the data as fresh as possible - also possibly keeping part of the allowance for forced refreshes.
Now, the main problem I see with this is cost of hosting not just website, but also a background running .NET application. But let's assume that the cost is not that relevant. What I actually fear is that I overcomplicate the solution. I might just put more of the website in Ajax and just wait for the request to complete in the background (when calls become available).
So, does anyone has any simillar experience? I'm especially interested in the scenario where it is all scaled up, that is we can assume that there will always be deficit of available calls.
PS I also worry about keeping website (which can be triggered independently by many users) from trying to refresh the same records twice etc. - I guess I should put some additional safeguards, lock or at least mark the records undergoing refresh etc. - a lot of code just to solve the problem that wouldn't exist with a single-backend....