Data pipeline with fallbacks and callbacks

Question

I'm refactoring our current design for how we download static data. It's a mess of deep class hierarchy and callback hell and I want to convert it to a more elegant straightforward design.

Here are two examples of the steps we take to download static data (JSON files):

Example one, downloading main manifest:

• Attempt to load manifest file from local disk
  • If fail, download from the internet
  • If fail to download, bail with error
  • If success, return string data
  • If success, deserialize manifest file
  • For each file in the manifest, attempt to load from local disk
    • If success, deserialize and return
    • If fail, download from the internet
    • If download succeeds, return string data
    • Deserialize and return
    • If download fails, bail with error
  • When all files are deserialized, return all results in a list

Example two, download a specific file:

• Ask manifest manager to get a specific version of the file
  • If manifest manager has it stored in memory, return
  • If manifest manager doesn't have it stored locally, attempt to download it
  • If download succeeds, deserialize and return
  • If download fails, bail with error

Currently, every step is a method in... some class in the hierarchy, and subclasses override some steps to accommodate some special casing (e.g. don't use manifest manager, don't attempt to donwload, etc).

Almost every step uses callbacks due to the nature of downloading stuff from the internet. And so, we ended up having to even override the callbacks in... some classes in the hierarchy, which resulted in a complete, unreadable mess.

My first attempt to refactor this was to use either the Decorator or Chain of Responsibility design pattern. The thing is, some steps require CoR and some don't. Moreover, some (if not all) steps use callbacks, which doesn't feel like the patterns where design to use. Yet another problem was that some steps in the chain would transform one type of data (string) into another (deserialized object). And finally, I have no idea how I would translate "take the downloaded manifest, and yield each file separately, and when everything ends, continue the chain".

This is the closest I got with a mix of Decorator and CoR classes, but it does not include the last part where we split and download each file individually.

var callbackReader = new CallbackGameDataReader<ManifestVO<ManifestFileVO>>();
var manifestDeserializer = new DeserializeGameDataReader<ManifestVO<ManifestFileVO>>(callbackReader);
var wwwReader = new WwwGameDataReader(remoteEnvironmentPath, manifestDeserializer);
var streamingReader = new StreamingAssetsGameDataReader(streamingAssetsManifestPath, manifestDeserializer, wwwReader);
streamingReader.Read(dataId, o =>
{
    ManifestVO<ManifestFileVO> localManifest = (ManifestVO<ManifestFileVO>)o;
    localParsedData = localManifest.files;
    localKeyedData = CreateKeyedData(localParsedData);
    parsedData = localManifest.files;
    keyedData = CreateKeyedData();
    ContinueInit();
});

It also doesn't look very intuitive.

My goal is to end up with something like this (kinda):

List<ManifestFile> dataFiles = new DataReader()
    .ReadLocal() // How do we fallback to WWW?
    .Deserialize<Manifest>()
    .Split(manifest => manifest.files)
        .ReadLocal()
        .Deserialize<ManifestFile>()
        .Merge() // ???
    .ToList();

I've been reading about Chain of Responsibility, Decorator, Future/Promises, Pipeline and trying to wrap my head around them. Part of the problem is that I've only done basic collection manipulation with Linq (where, select, etc) so something like map/reduce is still new to me, and the other part is that I'm not quite sure how callbacks fit with any of this (except maybe Future/Promises).

And then there is the fallback part. I could have a single GetFile that does attempt to do everything locally, but then, in cases where the process should always use local data, I have to duplicate logic.

So this is where I'm at now. I've been reading a bunch of patterns and trying to figure out how to decide which one best fits my needs and to glue them together.

My goals are:

• It needs to use callbacks
• Ideally, I can configure fallbacks (where appropriate)
• Should be type-safe (probably use generics)
• All steps should be (relatively) interchangeable (e.g. wwwReader with localReader)

I'd love to hear your thoughts. Any recommended material to read would be great.

Thanks!

P.S. Apart from actually solving the problem, my intention is to learn how to implement this from scratch, so I'd rather avoid using existing libraries

P.P.S. The solution needs to run in Unity, which uses a custom Mono runtime, so async/await is not an option. And using libraries might introduce issues when compiling for mobile devices.