Design Pattern or suggestion for implementing

Question

The Problem

A problem I keep running into again and again in C# is how to treat a class of things as the same in some contexts, but as different in different contexts. I have found some ways of handling this that I'm satisfied with for now, but this particular case I'm facing is leaving me unsatisfied.

I'm sure that defining an interface is the right way to go, as this allows me to treat very unlike things (nouns) as alike due to the verbs they expose (the interface implementation). But when at some point those things need to behave very differently, it becomes less clear to me. I'll try to give a concrete example and hope that you can suggest a way to design this well.

The Application

The main application is essentially a workflow engine. It works with 3 different types of main objects, and has interfaces for:

1. Reading those objects in the ubiquitous language of the application (despite differences in different data sources) through implementations of IDataSource.
2. Creating a strongly-typed representation of those objects via specific consumers IConsumer which return IConsumerMessage items.
3. And turning those strongly-typed consumer messages into mostly string-typed IRecipientMessage representations in a kind of transport layer IRecipient (where the final step of serializing to CSV or XML or whatever can be performed by common libraries, so the recipient interface implementations don't need to concern themselves with the guts of this conversion, though they specify what is desired).

So to summarize, there can be many data sources, one or more of which can produce application objects of type ObjectA, ObjectB, or ObjectC, which are then consumed by one or many IConsumers that each translate them into a particular IConsumerMessage representation, which are finally delivered to the actual real-world consumer companies these represent through a proximate recipient (a.k.a. a transport middleware) through one or more thin IConsumerRecipientMessage classes.

I'm happy with how all this is working. But now I need a generic way for some data lookups to be performed on the core application objects, and to manage the set of data needed by particular consumers.

The Workflow Prompting This Question

Say TrifidConsumer has special rules for which of the ObjectA types it wants to see. There is a daily list of ObjectA model numbers that the Trifid company is interested in for each day, available by FTP. So at an appropriate step in the workflow, we need to take each core ObjectA, look up the date and model number in our TrifidObjectADesirability data that we've downloaded, and keep track of the resulting yes/no flag, for later consumption by the TrifidConsumer implementation of the IConsumer when deciding what ConsumerMessage items it will return to the core of the system.

How This Workflow is Handled Now, and Desired Requirements

I'm refactoring an existing codebase, which currently is doing special logic in the core of the system on ObjectA types if it knows the consumer needs the special Trifid metadata. However, I think the application engine shouldn't know about the TrifidObjectADesirability data, and shouldn't know the difference between one consumer or another--this behavior should be able to be specified in a modular way, and be discovered or composed through some means so the engine is agnostic.

Additionally, on top of the normal workflow UI for processing objects of the different types, I need to present to the user a special management page where he can look at all presence or absence of loaded TrifidObjectADesirability data, see which dates are present, perhaps even manually look up a few values by model number, delete a particular date's data, force a fetch of a particular date, attempt a direct FTP pull of a date's data, upload a zip file of a date's data, or load a zip file from a predefined folder where a process has theoretically already downloaded such FTP data.

It is also possible that another consumer could want access to the TrifidObjectADesirability data and signal a desire to take a dependency on it (rather, on the result of its lookup for ObjectA objects), requiring a separation between the operation of the lookup and the consumption of it.

Finally, each new consumer that gets added could have its own set of data--perhaps instead of model number, it needs to perform some other lookup against color or size or some other attribute. It needs to have a set of data that is handled almost identically to the TrifidObjectADesirability data, but is different in its use and shape: It could be a lookup based on two fields on ObjectB items, that returns 3 fields that will need to be inspected later. Or this looked-up data could need to be hit against yet another data source before a final value can be determined.

The Actual (tl;dr) Question

What is a pattern that could help me accomplish these summarized objectives/requirements?

1. Define many data sets and metadata for each one's management (along with UI tools) such that they can be treated generically by the core application (though using supporting classes from the core application is fine, as long as the proper nouns and the individual details of the different data sets are not accessed in the core application via special knowledge of the data set).
2. Those data sets and attendant methods and metadata need to be combined into some kind of Enrichment container object.
3. The Enrichment data set container object needs to expose generic methods for fetching data (both from the external source, and from the saved local store), removing data, and deleting portions of it based on different criteria, returning up to a million rows for a whole day's data, or hitting the primary data source to (re)load a day's data on demand, storing looked-up data into a permanent local store, or looking up just a few values' presence in the data. These methods nonetheless need to be able to be used in the specifics ways they are different where appropriate.
4. Each IConsumer can indicate it has a dependency on the result of one or more of these data sets.
5. in the workflow that can provide the needed data, so that each IConsumer, when it receives each ObjectA, receives it with the required enrichments present so it can perform the enrichment (ask the concrete TrifidObjectADesirabilityEnrichment if each ObjectA is present in it), or the core system has already (generically) performed the enrichment and the result is presented within the ObjectA itself in some kind of dictionary or data structure acessible by the IConsumer.
6. All enrichment actions need at least temporary caching so if another IConsumer asks the same question (perhaps Dufflepud company says it wants to use the same Trifid data, it can, and whichever corresponding IConsumer is handed the immutable ObjectA first actually triggers the lookup a single time) it can be answered again.

Final Thoughts

One architecture with some similarities that comes to mind is ASP.Net MVC areas. They are like mini bolt-in web sites, that function in the context of a larger web sit as an integrated part of the whole that an external consumer can't tell was modularly applied, simply by being called during route and view-engine location registration.

P.S. I realize this question is very long, and I welcome ideas on how to make it shorter while still explaining the problem well enough to get a useful answer. I think this is an at least somewhat complex/tricky problem or I wouldn't have had to lay it all out like this, and I am very grateful for the time and energy of anyone who bothers to try to comprehend such a wall of text and make useful suggestions in response.

P.P.S. Solution dependency diagram (assume a transitive dependency throughout, except the web site and service do not depend on the DataSource, Consumers, or Recipients).

           Application
               ^
               |
            Business
               ^
      _________|__________
     /                    \
DataSource (1..n)     Consumers (1..n) & ConsumerMessages
     ^                    ^
     |                    |
     |                Recipients (1..n per Consumer) & RecipientMessages
     |                    ^
     \____________________/
               |
        CompositionRoot 
               ^
      _________|__________
     /                    \
     |                    |
 Web Site          Windows Service (perform automated tasks)

The dependencies are upward, in the sense that, e.g., the DataSource project has a reference to the Business and Application projects. The Business project gets its instances of IDataSource via the CompositionRoot, which for example instantiates each DataSource for this invocation of the application and passes them in as dependencies to the Business engine's main class constructor (and the same for each consumer and its paired recipients). A unit test can compose a different application, such as one with mocked dependencies, so it can actually unit test the core engine's behavior.

The composition root creates all needed dependency instances and combines them returning a composed Business-Application to the user of it (the web site, or windows service, or some future API or Windows Forms program or anything at all) for use. This can loosely be called poor man's DI, though not the kind that isn't DI at all (where you do new Dependency() inside of the class that needs it)—all dependencies are instantiated in a single, tiny class in the composition root whose only purpose is to compose the application and return it. The web site's IOC container/library can inject this composed application into controllers as necessary, but the service can also acquire the exact same composed application without having to repeat the construction of it via its own IOC container or other method.

I am thinking that I need to specify an IEnrichment interface in the business or application projects, somehow define how each different enrichment works in the DataSource or another (new) project, uniquely identify each concrete Enrichment via a ubiquitous language term such as an EnrichmentIdentifier that wraps a string indicating the ID of the enrichment, specify dependencies within each Consumer using an equatable instance of the same EnrichmentIdentifier string, and finally perform all these actions as described in the workflow above.

But I really am not sure, and sans working directly with folks who are experienced enough to help, I have turned to the P.SE community for help.

Answer 1

Your mindset is interfaces and it looks like that served you well so far. I am actually fairly impressed with the displayed analytical approach. It also seems you are stuck in this "interfaces can solve anything" dogma.

What I read is the need for a filter. An abstract base class Filter that implements IConsumerFilter with method Process(Object object) returning Object. If the returned object is null, there is no interest in the object. A non-null result will be the enriched object for the particular consumer. Each type of consumer gets its own Filter descendant that knows what type of input object it can expect and what type of result it should yield. The caller and receiver can validate the type and cast before proceding.

Who is to decide if an object is interesting or not? You, the system integrator? People at the consumer's site? Whoever this is needs to implement the Filter descendant or at least specify the criteria. The filters may be implemented in a separate assembly. Are the filter criteria fixed, can they be hard coded? Filters may need to be updated on a regular basis or be dynamic, they could read a piece of xml that specifies the filter. When that is updated, the filter could re-initialize itself.

I hope this helps or may trigger a better idea to emerge.