Supplying application state to a resuming Hangfire background job

Question

If this is better suited for StackOverflow let me know and I'll head over there... I think it belongs here though.

I'm writing an ASP.NET Core REST API that returns 201s and 202s after kicking off long running asynchronous jobs using Hangfire. If the jobs fail mid way Hangfire will pick things back up, run retries, etc. Obviously in order to do so, it needs to know some state.

Hangfire recommends that the arguments to background jobs remain "small and simple", and that if a background job needs some more complicated object to write the ID to a database.

My background jobs need several API keys, passwords, some not so easy to serialize objects, and some YAML files loaded as config via an internal common configuration library. Because of the sensitivity of these data, I would rather not write them to the database. The options I see here are two:

1. Pass each and every piece of the larger objects as arguments to the Hangfire job and let Hangfire serialize them to its DB. For sensitive arguments like API keys and passwords, first encrypt them using some cert bundled with the app. The background job will have to decrypt them using the same cert after fetching them from the database when it begins running.
2. Set the objects to some public static property in a public static class at startup. The background job can just refer to that when it needs. For example:

// Startup code simplified for the sake of the example
{
    var envConfigs = new EnvironmentConfiguration( ... a bunch of random stuff ... );
    service.AddSingleton(envConfigs);
    EnvironmentConfiguration.Instance = envConfigs;
}


// Hangfire job

{    
    var result = BusinessLogicDoerMethod(EnvironmentConfiguration.Instance.BusinessLogicInformation);
}

Option 1 is annoying because we have to pass a lot of arguments, and also have to deal with the encrypt / decrypt logic.

Option 2 is probably fine, the danger I see here is if one background job mutates the EnvironmentConfiguration object, the other jobs and the rest of the app will also see that mutation. However, I can easily solve this problem by simply not doing that.

Answer 1

1. Pass each and every piece of the larger objects as arguments to the Hangfire job and let Hangfire serialize them to its DB.

That's ignoring Hangfire's advice to not do that. Hangfire is quite lean but if you give it big chunks to process you are going to make it choke.

Because of the sensitivity of these data, I would rather not write them to the database.

This heavily conflicts with your first suggestion.

If you don't want to write these to the database, then you also shouldn't want to write these to the hangfire database either. Yet that is precisely what you suggest.

2. Set the objects to some public static property in a public static class at startup.

Global scope is never the answer.

This completely throws out the possibility of concurrent jobs (fighting over the same static memory), race conditions (queueing a new job when the previous one is still running), and resuming your application after an outage (your static values will be lost).

While a static object might be acceptable in cases of universal readonly data singletons such as your EnvironmentConfiguration.Instance, since you also register that config object in your service collection, I would advise you to stick to using the injected config rather than trying to also use it via a global static singleton.

My background jobs need several API keys, passwords [..]

Store these keys securely and tell the hangfire job where to get the key, rather than the key itself. For example, in our Azure projects we rely on a key vault. In other projects, I've relied on a similar but custom-built key vault.

How to store these keys securely is a matter of context and necessity. Maybe a local config file in the application files suffices. Maybe you need something more rigorous like an actual key vault.

[..] some not so easy to serialize objects [..]

If the issue is passing eyes (i.e. people with db access), encrypting the serialized data will solve the problem at hand. You'll have to decrypt it when you retrieve it, but that's a negligible overhead cost if it solves the privacy issue while keeping the hangfire handling easy and lean.

You can also set the hangfire job up to clean up its encrypted data from the database when the job has completed. Or you could have your job mark it as completed and then set up a recurring hangfire job to clean the data that was marked as completed.

[..] and some YAML files loaded as config via an internal common configuration library.

Why load the file before you actually need it? Instead of giving the hangfire job the loaded file data, just provide the reference (file path, I presume) for it to fetch its data itself.

This is in effect no different from hangfire's "pass an id, not an entity" advice, but it uses the file storage instead of the database. That distinction is irrelevant in terms of keeping hangfire jobs lightweight.