1

Often enough, the terms data model and data format are used interchangeably, but here I disagree. Let's start with the simpler one, the data format. I don't know exact definitions, but the data format describes the layout of a piece of data, the meaning of the individual bytes. There are many formats available, more general ones being for example JSON and XML. However, one can represent the same piece of data in different formats. Short example - a location.

JSON

{
    "location": {
        "longitude": 41.25,
        "latitide": -120.9762
    }
}

And XML

<location>
    <longitude>41.25</longitude>
    <latitude>-120.9762</latitude>
</location>

So, the basis for those two is a common data model, which states that a location is comprised of two fields, long and lat, and both those fields are floating point values.

Now my question is, how can I formally write down such a data model, preferrably in a common format that is machine readable (preferrably - a standardized diagram is already better than a sheet of paper)?

XSD for XML and JSON Schema for JSON are mappings of a data model onto a specific data format. What I was hoping to find is one level higher, providing an abstraction of the data model to be consumed my multiple, different data formats. I don't design the application, I design the data model. If application A consumes a piece of data from a web service, JSON might be a good choice for a data format here. But an application B, which works with the same data, but only over the wire, might prefer to use a binary format. I hope it gets clear what kind of abstraction I want - not to model data for a specific format, but for all formats.

  • 1
    There are tons of available data model specifications, often called schema specifications. Grafical ones (UML based, ERM based), textual ones like XML Schema or JSON Schema, the DDL parts of SQL, several proprietary ones. Often, the data type definition capabilities of a certain programming languages are used to define data models. Pick whatever fits best to your tooling. – Doc Brown Mar 25 at 12:07
  • 1
    You are correct that XML Schema and JSON Schema are aiming specificially for XML and JSON data formats, but their usage is in no way restricted to this. For example, both are powerful enough for specifying a relational data model, is that's required. – Doc Brown Mar 25 at 14:58
  • Schemas are not data models. The document object model and the relational model are two examples of data models. Both Json and XML are representations of a document model. – John Douma Mar 26 at 3:33
2

Data format

A data format defines a specific syntax to be used to store or exchange data. This syntax is usually defined with the help of a grammar. Some examples:

  • Json syntax is defined using a railroad syntax diagram (left side of the official website) and an indented variant of a BNF like grammar rules (right side of the official web-site).
  • XML syntax is defined using a rigorous BNF grammar (the terminals are defined using regular expressions).
  • CSV syntax is defined by the IETF both with a textual description and with a formal ABNF grammar, yet another variant of BNF.
  • In telecom industry, many data formats are themselves defined using the ASN.1 syntax.

As you rightly pointed out, the same data can be represented in different formats.

Data models

The data model defines the semantic of the data independently of its representation. But semantic has many levels:

  • A first level is to identify data that belongs together and has together a meaning. For example 27, 2, 2020 are just three numbers, but if taken together it can represent a date, the 27 February 2020

  • A second level is to identify tight relations between such groups of data, which can be mapped to concepts in the real world A person can for example be defined by a name, a date of birth, and a place of birth.

  • A third level is to identify more relationships for example that a person may attend a course, and a course may be attended by several persons. Relationships can be annotated to express what the relationship means.

  • A next level is to identify generalisation or specialisation relations between concepts.

The data modeling can be abstract and completely independent of the data representation. Typically, you would use and Entity-Relationship (ER) model or UML models (class diagrams). The first is focused on data and is heavily used in the area of RDBMS. The second is used in object oriented design and allows to address not only data but also associated behavior.

The important aspect in a data model is also to map the data to its meaning. This is done using a dictionary that defines each entity or class, the attributes and the relations.

The data schema

A schema such as XSD or JSON-Schema are in-between data format and data modelling:

  • A schema defines valid combinations of elements and value constraints. It therewith extends the general XML or JSON format to make it specific to some kind of data.
  • But at the same time, defining the elements that belong together de facto identifies the level one and two of our data modeling: it defines relevant data types, that could be considered independently of the representation.

Personally, I tend to see schemas more as a data format definition, since one of their main use is to have a validation of some data. Furthermore, it often happens that in a schema you would put together interrelated data, which could be modelled as distinct entities with a strong relationship and that are only put together tactically in the data format. But I understand that a schema could be considered as modeling tool, since it nevertheless allows also to work with/define/document data structures.

Machine readable models

UML is not machine readable. ER neither. In fact, the mainstream modeling languages are graphical. They could be encoded using a machine readable format, but there is no standard.

So if you want to use a machine readable model, the schemas seem to be the best option. For very strurctured data SQL is of course an alternative of choice, and many tools exist that reverse engineer SQL to draw a model.

| improve this answer | |
3

There is no absolutely universal way to describe a specific data model, which is completely platform and system independent.

From a very high point of view, the UML could be used, which is a very common language to describe not only building pieces of software but also other architectural parts, however UML lacks your requirement of being machine readable, as it's mostly targeted towards human interpretation.

Mostly, when you architecture a system and your requirement is to document available data models, it's your responsibility to define the way and format how those data models are presented, along with defining the interchangeable data format used for data exchange.

Should you choose JSON or XML, you will need some form of formal documentation which tells other developers what to expect. In case of JSON, you could use JSON Schema, in case of XML, you could use WSDL. Should you be building a public API, regardless of the exchange format used, you could use a platform such as Apiary to provide the description of available operations, along their input and output data models.

The used approach depends on the kind of project you're building, and the chosen data exchange format.

| improve this answer | |
  • The main point is: I don't choose a data exchange format, that is up to the application. All I need is to define the semantics of the data. Some service might consume location data in JSON, some service might consume locations in binary format. But they all need to be aware that they can extract long and lat but nothing more from the data. – flowit Mar 25 at 12:24
  • @flowit, what is a JSON if not a string with certain semantics? What is a string if not an array of chars, which can be converted to binary data? By choosing an exchange format, you state what kind of semantics is expected. Whether someone consumes binary data or not, the semantics coming from the used format must be adhered to in order to correctly process the input. You either call some source to retrieve data, and then must follow the source's semantics in order to correctly read the data, or you are called by something and in such case it's necessary your rules are followed by the caller. – Andy Mar 25 at 12:55
  • 1
    @flowit, what you're describing sounds like some sort of "intermediary format", which will have to be chosen or specified to be compatible with a specific set of final formats. There is no universal format for the representation of data, because there are too many ways in which all sorts of data can be represented, not all of which are compatible. Representations like JSON and XML are already highly generic. – Steve Mar 25 at 12:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.