-2

I am looking to an IoT like system which must deal with a wide range of data complexities. As an example, I'd like to model a 'thing' and I know all things have a location. However, when the actual data arrives from these things (from different vendors), something like latitude can be spelled, it may have different naming standards (vendor1=lat, vendor2=latitude). I want to build an abstract model at the application layer which knows how to manage these gaps. Preferably using an XML data representation (ie, I could use a std like OAGIS/MIMOSA).

Couple of questions:

1) Has anyone done something similar?

2) Seems like it could be done at the data ingestion layer (ie, harmonize all data into 1 def of lat) or it could be done at query time (when someone executes an API, find the right column and return it). Has anyone done a comparison of these two models?

closed as off-topic by DeadMG, Telastyn, Jörg W Mittag, gnat, user22815 Jan 28 '17 at 19:33

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions asking us to find or recommend tools, libraries, programming languages, resources (including books, blogs, tutorials, and examples), or projects to undertake are off-topic here as they attract opinionated answers that won't have lasting value to others. You may be able to get help in The Whiteboard, our chat room." – DeadMG, Jörg W Mittag, Community
If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    These sound like recommendations for opinion polls (the people on the site) or off-site resources (external comparisons). – DeadMG Jan 27 '17 at 19:04
2

Having done a couple of similar things, not IOT, I would say that the data ingestion model is preferred. When the data arrives you know where and what it is coming from so that is the natural stage to map the data into a uniform format. Otherwise you will need to keep track of exactly which sensor/input/source the data came from, (not too bad a thing in any case), have a non-heterogeneous storage type, re-parse & transform each data element in the even of things like requesting a sorted result, etc.

Thinking about your example there are numerous formats for transferring latitude & longitude between devices and if you needed to group all of the samples from within a specific geographic zone your processing requirements grow enormously if you have to convert each pair of readings as well as compare with the bounds.

From the cynical, or experienced, point of view I would recommend, if space permits, storing the raw data and the source types as well. Several times I have had months of data that could only be duplicated at great cost, e.g. several months of "bake and shake" test data that it was discovered, retrospectively, had been captured with incorrect transfer functions. Having the raw data allowed us to re-transform the impacted readings and re-assess the results.

1

If the data coming in from your different sources is in XML, you could have a set of XSL transforms (probably one for each distinct input source) that format all the data into a single, common format that conforms to an XML schema that you define. I've done something similar, it worked for me.

If some of your incoming data is in some format other than XML, you'll either need to transform it into XML somehow, or find some other way entirely.

Not the answer you're looking for? Browse other questions tagged or ask your own question.