We need to create an application that must diff relationships between entities.

As an example, let's say we're creating an application to model instances of a disease amongst pets, and we periodically collect data from multiple XML sources.

We have some model that represents pets and one to represent diseases. We are interested in examining the change in 'joins' from pets to disease each time we collect the XML data.

It is insufficient to simply diff the XML, as we are using multiple sources and must have our own intermediate model.

I don't know if there is a way to do it, but I guess we could diff the pets_diseases join table if we were to build this on an RDBMS?

I also don't know a lot about graph databases, so would welcome suggestions. If this is the right way to go Neo4J looks like a credible candidate.


I think it depends on how sophisticated you want to get with the collected samples. You might start out collecting samples at some interval and diffing them, but over time, I suspect you will get more sophisticated realizing that your collected samples from the different sources are coming in more staggered.

As you proceed you might eventually see that there is benefit in collecting a date/time for each individual row of the data because even within one data collection from one source, the snippets of information have time variances.

I'm suggesting that you push time information as far into the system as possible, including both deep into the "front end" in collection process with XML model (individual rows being dated) as well as the intermediate database model (same). This will allow all the data collected to co-exist in the database, and will offer opportunities to run richer queries.

pet-table: id, pet-name, start-date, end-date
pet-condition: pet-id, condition-id, info-capture-date, status
condition-table: id, description

Given the pet-condition info-capture-date, you can store multiple data time points for the same condition (and/or multiple conditions for the same time).

Have a look at the following article for some food for thought: Time Series Database.

The format I'm suggesting is also log database friendly, or event store friendly, because it is essentially immutable (write once/append-only). However, It doesn't directly store just the current status, so you have if you want that you can write a query for it. (And if that is really what you need, you can cache that in the database).

Though most databases will handle log-type data, if your data collection becomes automated and a pretty heavy stream at that, there are specialized databases designed to handle log-style collection, such MongoDB.

I'd look into a graph database if you have many, varied, and dynamic (in the sense of the schema) relationships. Otherwise, I'd probably try to make do with a relational database.

Not the answer you're looking for? Browse other questions tagged or ask your own question.