1

I have an MVC 5 ASP.net app that is working perfectly. A colleague of mine has made a project that reads in data from files and churns some of it to output some calculations for me to display in the web application. I call his methods and I get the data back and display on screen.

As part of his project he has implemented a kind of caching system, it's not using any of the caching tools in. NET, it's simply holding some of the files required for calculating the numbers in memory as these take a long time to load in and are definitely the bottle neck.

So I need to do two things

1) Have one instance of the class within the project so it makes use of the in memory files.

2) Setup a worker process that runs at 7am everyday to load those files into memo.

For number one I have instantiated my colleague's class as a static class and this seems to work some of the time but it's unreliable for some reason I'm guessing due to app pool timeouts, I'm also worried about the effectiveness of this and whether or not I will have issues with multiple users sending concurrent requests. Is there a better way to achieve this? I've looked at various .NET caching libraries so I could get my colleague to implement a proper cache but for now I'm assuming I can't ask him. I'm a little confused as to why in MVC every time a user sends a new request a new controller is instantiated hence instantiating my colleague's class again and losing the in-memory files?

For number 2 I've read about webbackgrounder a library for ASP.net that can run background tasks. All I need to do is whilst the app is running on IIS then run a method everyday at 7am.

Update:

I can't save the output because I need to calculate these vlaues on the fly, there's about 10 different inputs into the calculation that can change to be any number of different values.

I can indeed put the input files into a database but that doesn't solve my problem, it wouldn't be any faster than loading them in from files, which takes too long as it is, I want to hold these items in memory hence the cache approach.

  • 1
    Is there any reason to not store the calculated results in a DB, or in a file somewhere your web app can get to it? It sounds like once they are calculated, there's no need to recalculate them. But please absolutely do not implement a task scheduler running under Asp.Net.. – Andy Jun 25 '17 at 21:15
  • just load it in app start – Ewan Jun 26 '17 at 8:08
  • @Andy it's the loading of the files that takes so long, and if I have my web app get to the files I then couple quite tightly the class library of my colleague's work and the web app which I really don't want to do – Simon Nicholls Jun 26 '17 at 9:21
  • @SimonNicholls You said your coworker produces some data from the loaded file; my question is why not save your coworkers calculations in a separate file or database, and just display it? – Andy Jun 26 '17 at 21:54
  • @Andy ah I see yeah sorry the calculations need to be done on the fly, there could be any number of combinations of inputs that change what the outputs would be so storing them is not viable – Simon Nicholls Jun 27 '17 at 9:07
1

Deploy your colleague's library as a cached microservice

Assuming you can't change your friend's library at all, you don't want to use a database, and you want to maintain a reasonable level of process isolation, here's the right way to do it:

  1. Write a bit of glue to expose your colleague's library as a RESTful microservice. If you've never done this before, start here. It shouldn't take more than 100 lines of code to wrap one function call.

  2. Use the uniform interface to manage cache and its duration.

  3. Set the application pool for this service to recycle at 6:59 am.

  4. Modify your web site to utilize the new service.

  5. If your colleague's code takes a long time to run, the microservice may not be responsive when handling its very first request. To deal with this, you need to "warm up" the service. Set up warmup using IIS auto-start; details can be found here.

With this approach, your colleague's code is completely isolated from yours, and the caching logic simply leverages standard features in IIS. With the caching and recycle scheme set up properly, you should have seamless transitions when the program is recycled, since IIS will continue to serve requests in flight from the old cache while building up the new cache. We set the cache expiration to exactly 7:01 am (as opposed to letting the app pool recycle take care of it) just in case there is some other network node looking at your cache headers, e.g. a proxy server that might cache the response if it isn't encrypted.

  • This is by far the best answer, and the one I'm going to try implement thank you! – Simon Nicholls Jun 27 '17 at 9:25
2

ASP.NET MVC (on IIS) cannot be used reliably for any sort of background work. The problem is that IIS processes are recycled regularly, and you can never be certain that a background task fired off will finish correctly (this is due to various reasons inherent to the process and the environment in which it operates). There's no such "fire and forget" functionality that can work well with ASP.NET MVC (or anything else running on IIS).

What I've done in the past for very simple timed tasks is to simply create a little console application that taps into the main project and place it on the server to run via Windows Task Scheduler.

For (slightly) more complex scenarios, writing a "command" to a database, then having a server-based (console, typically) application run that polls for new tasks to execute on a regular basis.

For robust management, look into a queue system like RabbitMQ, which can effectively decouple the background process from the MVC process.

(at least part of the reason for the recycling of IIS processes/AppPools and the fact that a controller is instantiated for each new request is because it helps ensure that if one request "goes bad", it doesn't bring down the entire system)

Here is the first reply on a search for "ASP.NET fire and forget": https://stackoverflow.com/questions/18502745/fire-and-forget-async-method-in-asp-net-mvc

  • This is a possibility (running as separate app), what did you use to send data back and forth between the apps? – Simon Nicholls Jun 26 '17 at 9:26
  • Either toss it in a database field to be handled by the receiving app, or use a message broker like RabbitMQ. Depends on the complexity/robustness requirements. You could dump it to file and read it on the other side if you want as well, but the idea is that there's no inter-process communication, so you need something to persist it between the two apps. – jleach Jun 26 '17 at 10:34
  • I want the data to be in memory though and not in a database, it's a vast amount of data that I don't want to be throwing back and forth from a database, plus the data in the files never changes so it makes perfect sense to put them in memory – Simon Nicholls Jun 27 '17 at 9:10
0

ASP.NET comes with a variety of caching mechanisms. The best fit for this situation is probably HttpApplicationCache. It has a specific class for monitoring flat files and keeping the cache up to date.

Basically you would set up a static property that wraps the caching logic, and use the static property whenever you want to access the file.

static public string CachedFileContents
{
    get
    {
        ObjectCache cache = MemoryCache.Default;
        string fileContents = cache["filecontents"] as string;

        if (fileContents == null)
        {
            CacheItemPolicy policy = new CacheItemPolicy();
            policy.AbsoluteExpiration = DateTimeOffset.Now.AddHours(24);  //Set to taste

            List<string> filePaths = new List<string>();
            string cachedFilePath = Server.MapPath("~") + "\\cacheText.txt";
            filePaths.Add(cachedFilePath);
            policy.ChangeMonitors.Add(new HostFileChangeMonitor(filePaths));

            // Fetch the file contents.
            fileContents = File.ReadAllText(cachedFilePath);

            cache.Set("filecontents", fileContents, policy);    
        }
        return fileContents;
    }
}
  • So I want to keep my colleagues library separate from .NET, just as a simple class library as it might be used elsewhere. I feel like this is a bit of a flaw implementing HttpApplicationCache as it couples the class library very tightly to my web app then and I don't really want to do that. – Simon Nicholls Jun 26 '17 at 8:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.