We are using a third party REST API and we are building one of the pages on our ASP.NET site around the data it delivers. We get results in JSON, and interact with it mostly using .NET MongoDB Driver.
The server behind their API uses mongoDB, where the documents does not have to follow any structure. The problem is that the document structure of the JSON that I receive from their REST API is not consistent either. The provider says:
Sometimes we get incorrect data, that's just how this system works. We expose all our data in the API, even if it is inconsistent.
I am building a system around it, and if the data I base it on is inconsistent, I am not sure what to do. I am new to talking to a "NoSQL REST API", and would like to know the best approach to handle this situation.
My thoughts so far below. Which are right or wrong and why?
As a consumer, I must do null-checks, invent default values, or discard result items, for every single value in every single level in every document structure that somehow may be corrupted. This will take a lot of time for me and is hard to maintain.
Even if the Server runs MongoDB in the background, and does not itself have to follow structure, a REST API should still return JSON following a structure, and be able to tell beforehand what it will return. It should be up to the provider of the API to clean their data to conform to a consistent JSON structure, not the consumer.
Middle way: The provider and the consumer (me) has to beforehand agree on what values that should be always expected. A kind of skeleton JSON structure. This is what I would like, but I would probably not be able to trust it 100%.
I should use some architecture / pattern / component / solution that I am not yet aware of.
While a winning answer would solve the problem generically, or bring a greater understanding to the problem, I am looking forward to all kind of answers!