1

A client has requested that we build a platform for integrating data from partner to their central data store. This will not be "Big Data" scale. The data from each partner will be accessible through RESTful APIs. Since RESTful APIs are not standards based (at least not currently), I divide the problem into three parts:

  1. Authentication / Authorization
  2. Access Pattern
  3. Data Structure

I see each part as having it's own internal complexities. We proposed a "driver" model, where each API type will have a driver, which will be responsible for taking care of these three issues for each API individually.

If a new API comes along, we can develop a new driver for it.

The problem is that the customer does not want to require development resources for each new API.

I have been looking into ETL / Data Integration tools from the likes of Informatica, IBM, etc. and also some open source options. None seem to cover even a small part of the complexities of each.

I am looking for an approach that can provide the kind of solution the customer is looking for, but am having trouble finding one.

  • @Mat - Sorry, updated the title. It was auto-completed. Thanks for noticing :) – Elad Lachmi Nov 9 '16 at 16:35
  • 1
    So you want one bit of code that can deal with an arbitrary APIs potentially different Authentication Authorization and Data Formats? If this existed there would be a lot less work for programmers and I imagine a lot more errors in misinterpreted data! – Milney Nov 9 '16 at 16:50
  • The closest I can see you getting is defining code for each of the 'types' of API and having any differing things as metadata / configurable parameters – Milney Nov 9 '16 at 16:51
  • @Milney Thanks for your comment. I want to create software that matches our client's requirements. We know they are unrealistic, and that is why I'm looking for the next best thing. – Elad Lachmi Nov 9 '16 at 17:17
  • 1
    This doesn't seem possible, unless the client has vast monetary resources to develop such a platform. What you're describing is called Middleware. – Robert Harvey Nov 9 '16 at 18:02
4

The fundamental problem here is that REST is not a standard. It is an architectural style for web APIs. While there are mechanisms to make these APIs self-describing, and there are web-service description formats, most APIs do not make use of these techniques – and why should they? That's a lot of enterprisey ceremony with very little value.

Even once you can represent the structure of the API, you still have to assign a meaning to the data you are receiving. That's not really something a computer can figure out for itself. You'll want to look at the API you're adapting, and write the mapping yourself.

The good news is that this mapping from received data to your customer's datamodel can often be quite simple, to the point that you may be able to specify it in a configuration file. That's still a kind of programming, though. Most importantly, these configurations need to be tested very thoroughly to avoid data loss and data corruption.

If the APIs and your datamodel are all based on XML, you can use XSLT to specify most of these transformations. I'm sure a similar transformation technology exists for JSON, but it would seriously be much simpler to do the transformation in JavaScript.

Since there is no one-size-fits-all solution to these problems, it's not surprising that you didn't find any suitable existing product. This is a classic case where custom software development is unavoidable. The funny thing is that it's probably cheaper and quicker to just write one adapter per API (and re-use common parts as the development team notices similarities among APIs), than it is to develop a generic adapter that can deal with anything. Such vague, overcomplicated projects carry significantly more risk, and might not ever deliver any value.

What options does that leave for you? Ideally, you'd be able to convince the client to discard this dream of an universal API adapter. You can try to dress up the separate adapters as an “integrated platform” that can be extended to new APIs with “minimal” configuration code. You could write a generic adapter that is able to handle the currently needed types of APIs, but would need further development if other types are needed. That's probably a good compromise. But whatever you do, don't enter a fixed-price contract to develop that utopic universal driver – that will only ever exist in your client's imagination.

  • 1
    XSLT? Eeek. See also stackoverflow.com/q/1618038 – Robert Harvey Nov 9 '16 at 19:42
  • Thank you for the detailed answer. That was also my understanding more or less. Another option (which is the one we decided to pitch) is to use a declarative method of building adapters for each new API, as opposed to imperatively writing new code for each new API. Then we could create a nice GUI so that any user could create the new mappings. – Elad Lachmi Nov 10 '16 at 3:52
0

It's hard to give a solid answer without knowing what the rest end points are going to be used for. However, I assume the client simply want's to dump data into a thing and then be able to look at it, without having to write a bunch of code. If that data is not binary, all you have to do is stand up something like Elastic Search, which will take pretty much anything you can convert to JSON and has decent free web based tools to search and visualize the data. I realize that drastically reduces the amount of software engineering involved, but it could solve your client's problems.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.