I'm having some trouble finding a proper solution to the following architecture problem.

In our setting (sketched below) we have 2 data sources, where data source A is the primary source for items of type Foo. A secondary data source exists which can be used to retrieve additional information on a Foo; however this information does not always exist.

Furthermore, data source A can be used to retrieve items of type Bar. However, each Bar refers to a Foo. The difficulty here is that each Bar should refer to a Foo which, if available, also contains the information as augmented by data source B.

My question is: how to remove the tight coupling between SubsystemA.1 and DataSourceB?


  • 11
    That's a beautiful sketch. What program did you use to draw it?
    – Marcelo MD
    Nov 16 '12 at 16:48
  • I also want to know what program did you use to draw it. Please. Nov 16 '12 at 17:54
  • 2
    yuml.me is the site he used more than likely to create the diagram. Nov 16 '12 at 18:02
  • 1
    Aren't DataSourceA and DataSourceB already decoupled ? DataSourceA has a dependency on both SubSystemA.1 and SubSystemA.2, but not on DataSourceB. Nov 16 '12 at 18:03
  • 1
    @fstuijt No, it's not. If you modify SubsystemA.1 to use something other than DataSourceB, DataSourceA wouldn't know. DataSourceA only cares that SubsystemA.1 has a getFoo(id) method. There's an abstraction between DataSourceA and DataSourceB. Nov 16 '12 at 20:07

I created an app with much the same data architecture behind it; we have an onsite SQL database containing most of the automation and internal day-to-day information, and then a third-party cloud service used for sales, account management, field personnel, etc. Helpdesk needed information from both regarding clients' physical locations and equipment, and had been getting it from two different applications until I stepped in.

The long and short is that one data source needs to have a reference to the records of the other. In our case, the third-party cloud data contains references to the onsite data because that's the arrangement we had the most control over. Now, with an ID for a record from either data source, we can get data from both; with a cloud ID, we pull the record from the cloud, get the onsite ID, and pull the onsite data. With an onsite ID, we poll both data sources based on that ID.

In my system, I didn't make either object a child of the other in the domain layer; any usage of the data from both of the stores must maintain two object instances. Neither one is guaranteed to exist, which is why I did it that way; the app can work only with cloud data, or with onsite data, or both, with more limitations the less data it has.

However, that's not difficult to change, especially if you are sure that one side will always exist; simply include a property in the object representing the side for which data will always exist, that is of the object type representing the other data store's record. More advanced "merging" of the two graphs into one is possible.

This sort of arrangement must necessarily be coupled at some level. You can have a DAL that can interface with both data stores, or you can segment the DALs, one per data store, and have a higher layer such as a Controller get the data from each and snap them together. But, at some level, your program has to have the smarts to put these two disparate data sources' data together.

You can reduce the coupling required in most cases by abstracting away details of exactly where the data comes from. If you get data from a web service, which is given to you as instances of generated classes, then put a converter in place to make a deep copy of the service class into something you control, which won't have to change if the data source does (only if the schema does).

Now, this can be a huge undertaking; the cloud we use has dozens of domain classes, some of which have hundreds of data fields, and - here's the kicker - you could very easily have to make large changes to the abstract data type to accommodate a move to a different cloud or other remote data source. For that reason, I didn't bother; I use the generated web service domain directly, and now that a change from the cloud to an offsite (but under our control) data store is looming, the details of which I still do not know, I am simply planning to change the forms and codebehinds of the app, which is where the data is "combined", to reflect the new schema and/or data objects. It's a big job whichever way you slice it.

  • This answer best covers the problem that I've encountered, and in my opinion also offers the best answer so far. However, I would think that combining data from multiple sources is a common problem . Any design patterns that might help?
    – fstuijt
    Nov 16 '12 at 20:41
  • 1
    Some variation on a Factory pattern could be useful. If you have a CloudInvoice and a SqlInvoice object (from their respective data sources) and you want to create a single unified Invoice, create a Factory that knows enough about both data sources to retrieve each half of the record it's about to create, and then merges the information into your domain class.
    – KeithS
    Nov 19 '12 at 15:28

One way to deal with this is to make an aggregated data source that contains the data from both data sources in one place. A job would run periodically to check for changes in sources A and B, and writing the "deltas" into your aggregated data source. This would convert two tightly coupled data source into a single coherent data source.

Several things could prevent you from taking this approach:

  • The amount of data may be prohibitive - A full copy of A and B needs to be made, doubling the space requirements.
  • The data mus be live - There will be periods between the time the data in the source has changed and the aggregating job has propagated it into the aggregated source.
  • You need to reconcile the data with the original sources - The task of moving the changes back into their original places becomes a lot more complex if you take this approach.
  • I agree, introducing an abstraction layer is the preferred approach
    – neontapir
    Nov 16 '12 at 15:49
  • 2
    You can have an abstraction layer without copying the data.
    – smp7d
    Nov 16 '12 at 15:54
  • @smp7d This would hide the coupling behind a nice front end; I assumed that you were already doing something like that in your system, though, because otherwise the complexity would be spread across your entire design. Nov 16 '12 at 16:25
  • Depending on the DB environment this could also be handled with one/more Views, removing the need for copying data.
    – Walter
    Nov 16 '12 at 16:35
  • Aren't DataSourceA and DataSourceB already decoupled ? DataSourceA has a dependency on both SubSystemA.1 and SubSystemA.2, but not on DataSourceB. Nov 16 '12 at 18:02

It seems that at the top level there are two types: Foo and Bar, and you have only two top-level actions: findFoo(...) and findBar(...). That is the interface to the I/O layer.

Your description of the data sources implies that there are two methods on A: findFoo and findBar and one method on B: findFooAuxiliaryInformation. In findFoo you will need to merge the information from A and B.

I'm not sure what "tight coupling" you are referring to. There are three data types contained in the two datasets: Bar, Foo, and FooAuxData. The coupling between Foo and FooAuxData is inherent in the input data, and cannot be reduced. But that coupling should appear only in the findFoo method. That is the best you can do. The requirement is implemented in a single place. If it changes, you have to change that code.


You can't.

If I understand correctly, Foo and Bar come from dsA. Bars belong to Foos.
Preferably, you don't want Bars assigned to Foos, unless the Foo has been supplemented by Foo.enhancedInfo that comes from dsB.

Your preference for assigning Bars to Foos is what's creating your tight coupling. I would qualify that as a "requirements challenge" which is forcing you down a particular path.

So the technical challenges are that dsB may or may not have information on any given Foo and that dsB may not even be available.

You need to decide how hard and fast that preference for Foo.enhancedInfo really is. Based upon that requirement, you can decide to either provide a Foo + Bar object, or not. Allowing a non-enhanced Foo to be provided only complicates the logic and tells me that the preference isn't as strict as it may appear. Determine what variants of Foo, Foo.enhanced, and Bar your application(s) can support and you'll have your ultimate answer.

There are other things you could do to move the Foo related information closer together, and that might solve some of these problems. The way your question is phrased, it sounds like you're dealing with the problem at the data object level and you may not be able to consider infrastructure type changes.

  • The other way 'round: Foo s belong to Bar s
    – fstuijt
    Nov 16 '12 at 15:46
  • @fstuijt I'll update my answer in a bit. Fundamentally, it will remain the same. You need to decide how you want to server up Bar + Foo's.
    – user53019
    Nov 16 '12 at 15:50

If the data in data source B cannot stand on its own, you would probably want to migrate it over to data source A if possible.

If they are independent but related, you should look into data virtualization. This will allow applications to treat the data as one set (when appropriate) in an agnostic manner. Depending on your platform, there will probably be an existing framework/library that can help you implement this.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.