The Problem
A problem I keep running into again and again in C# is how to treat a class of things as the same in some contexts, but as different in different contexts. I have found some ways of handling this that I'm satisfied with for now, but this particular case I'm facing is leaving me unsatisfied.
I'm sure that defining an interface is the right way to go, as this allows me to treat very unlike things (nouns) as alike due to the verbs they expose (the interface implementation). But when at some point those things need to behave very differently, it becomes less clear to me. I'll try to give a concrete example and hope that you can suggest a way to design this well.
The Application
The main application is essentially a workflow engine. It works with 3 different types of main objects, and has interfaces for:
- Reading those objects in the ubiquitous language of the application (despite differences in different data sources) through implementations of
IDataSource
. - Creating a strongly-typed representation of those objects via specific consumers
IConsumer
which returnIConsumerMessage
items. - And turning those strongly-typed consumer messages into mostly string-typed
IRecipientMessage
representations in a kind of transport layerIRecipient
(where the final step of serializing to CSV or XML or whatever can be performed by common libraries, so the recipient interface implementations don't need to concern themselves with the guts of this conversion, though they specify what is desired).
So to summarize, there can be many data sources, one or more of which can produce application objects of type ObjectA
, ObjectB
, or ObjectC
, which are then consumed by one or many IConsumers
that each translate them into a particular IConsumerMessage
representation, which are finally delivered to the actual real-world consumer companies these represent through a proximate recipient (a.k.a. a transport middleware) through one or more thin IConsumerRecipientMessage
classes.
I'm happy with how all this is working. But now I need a generic way for some data lookups to be performed on the core application objects, and to manage the set of data needed by particular consumers.
The Workflow Prompting This Question
Say TrifidConsumer
has special rules for which of the ObjectA
types it wants to see. There is a daily list of ObjectA
model numbers that the Trifid company is interested in for each day, available by FTP. So at an appropriate step in the workflow, we need to take each core ObjectA
, look up the date and model number in our TrifidObjectADesirability
data that we've downloaded, and keep track of the resulting yes/no flag, for later consumption by the TrifidConsumer
implementation of the IConsumer
when deciding what ConsumerMessage
items it will return to the core of the system.
How This Workflow is Handled Now, and Desired Requirements
I'm refactoring an existing codebase, which currently is doing special logic in the core of the system on ObjectA
types if it knows the consumer needs the special Trifid metadata. However, I think the application engine shouldn't know about the TrifidObjectADesirability
data, and shouldn't know the difference between one consumer or another--this behavior should be able to be specified in a modular way, and be discovered or composed through some means so the engine is agnostic.
Additionally, on top of the normal workflow UI for processing objects of the different types, I need to present to the user a special management page where he can look at all presence or absence of loaded TrifidObjectADesirability
data, see which dates are present, perhaps even manually look up a few values by model number, delete a particular date's data, force a fetch of a particular date, attempt a direct FTP pull of a date's data, upload a zip file of a date's data, or load a zip file from a predefined folder where a process has theoretically already downloaded such FTP data.
It is also possible that another consumer could want access to the TrifidObjectADesirability
data and signal a desire to take a dependency on it (rather, on the result of its lookup for ObjectA
objects), requiring a separation between the operation of the lookup and the consumption of it.
Finally, each new consumer that gets added could have its own set of data--perhaps instead of model number, it needs to perform some other lookup against color or size or some other attribute. It needs to have a set of data that is handled almost identically to the TrifidObjectADesirability
data, but is different in its use and shape: It could be a lookup based on two fields on ObjectB
items, that returns 3 fields that will need to be inspected later. Or this looked-up data could need to be hit against yet another data source before a final value can be determined.
The Actual (tl;dr) Question
What is a pattern that could help me accomplish these summarized objectives/requirements?
- Define many data sets and metadata for each one's management (along with UI tools) such that they can be treated generically by the core application (though using supporting classes from the core application is fine, as long as the proper nouns and the individual details of the different data sets are not accessed in the core application via special knowledge of the data set).
- Those data sets and attendant methods and metadata need to be combined into some kind of
Enrichment
container object. - The
Enrichment
data set container object needs to expose generic methods for fetching data (both from the external source, and from the saved local store), removing data, and deleting portions of it based on different criteria, returning up to a million rows for a whole day's data, or hitting the primary data source to (re)load a day's data on demand, storing looked-up data into a permanent local store, or looking up just a few values' presence in the data. These methods nonetheless need to be able to be used in the specifics ways they are different where appropriate. - Each
IConsumer
can indicate it has a dependency on the result of one or more of these data sets. - in the workflow that can provide the needed data, so that each
IConsumer
, when it receives eachObjectA
, receives it with the required enrichments present so it can perform the enrichment (ask the concreteTrifidObjectADesirabilityEnrichment
if eachObjectA
is present in it), or the core system has already (generically) performed the enrichment and the result is presented within theObjectA
itself in some kind of dictionary or data structure acessible by theIConsumer
. - All enrichment actions need at least temporary caching so if another
IConsumer
asks the same question (perhaps Dufflepud company says it wants to use the same Trifid data, it can, and whichever correspondingIConsumer
is handed the immutableObjectA
first actually triggers the lookup a single time) it can be answered again.
Final Thoughts
One architecture with some similarities that comes to mind is ASP.Net MVC areas. They are like mini bolt-in web sites, that function in the context of a larger web sit as an integrated part of the whole that an external consumer can't tell was modularly applied, simply by being called during route and view-engine location registration.
P.S. I realize this question is very long, and I welcome ideas on how to make it shorter while still explaining the problem well enough to get a useful answer. I think this is an at least somewhat complex/tricky problem or I wouldn't have had to lay it all out like this, and I am very grateful for the time and energy of anyone who bothers to try to comprehend such a wall of text and make useful suggestions in response.
P.P.S. Solution dependency diagram (assume a transitive dependency throughout, except the web site and service do not depend on the DataSource, Consumers, or Recipients).
Application
^
|
Business
^
_________|__________
/ \
DataSource (1..n) Consumers (1..n) & ConsumerMessages
^ ^
| |
| Recipients (1..n per Consumer) & RecipientMessages
| ^
\____________________/
|
CompositionRoot
^
_________|__________
/ \
| |
Web Site Windows Service (perform automated tasks)
The dependencies are upward, in the sense that, e.g., the DataSource project has a reference to the Business and Application projects. The Business project gets its instances of IDataSource
via the CompositionRoot, which for example instantiates each DataSource for this invocation of the application and passes them in as dependencies to the Business engine's main class constructor (and the same for each consumer and its paired recipients). A unit test can compose a different application, such as one with mocked dependencies, so it can actually unit test the core engine's behavior.
The composition root creates all needed dependency instances and combines them returning a composed Business-Application to the user of it (the web site, or windows service, or some future API or Windows Forms program or anything at all) for use. This can loosely be called poor man's DI, though not the kind that isn't DI at all (where you do new Dependency()
inside of the class that needs it)—all dependencies are instantiated in a single, tiny class in the composition root whose only purpose is to compose the application and return it. The web site's IOC container/library can inject this composed application into controllers as necessary, but the service can also acquire the exact same composed application without having to repeat the construction of it via its own IOC container or other method.
I am thinking that I need to specify an IEnrichment
interface in the business or application projects, somehow define how each different enrichment works in the DataSource or another (new) project, uniquely identify each concrete Enrichment
via a ubiquitous language term such as an EnrichmentIdentifier
that wraps a string indicating the ID of the enrichment, specify dependencies within each Consumer using an equatable instance of the same EnrichmentIdentifier
string, and finally perform all these actions as described in the workflow above.
But I really am not sure, and sans working directly with folks who are experienced enough to help, I have turned to the P.SE community for help.