Best way to consume very dynamic/inconsistent xml/json

Question

I don't know whether to call the data dynamic or inconsistent. But I need to create profile pages for people generated from xml or json. The challenge is in the data the is returned. The data is bibliographical data about a person, and the APIs that are returning it are old, 11 years old. They need revisiting.

the data can look something like this

    <person>
         <personalinfo>
              //Always the same......
         </personalinfo>
         <categories>
             <publications>
                 <item> //<---The issues lie inside here.
                      <authors>
                      </authors>
                      <publications>
                      </publications>
                 </item>
             </publications>
         </categories>
    </person>

The issue lies with the categories and item information. A new category can be added at any time and be called anything. Also an item can have any fields and those fields can have any name and be added at any time. Essentially I don't know what I am getting back. On top of that there is no token returned giving any hints about displaying the information nor is there any format requirements on data that is entered into the feed that these APIs are returning.

I know that these APIs need updating but that isn't on the table right now. I just got a deadline pushed forward 2 weeks and need to have profile pages done very soon.

Are there any good tools that can handle this mess of information? Does anyone have any suggestions for getting this done quickly. This is going to be an iterative project I imagine so whatever I use is most likely going to be an interim solution. The data returned can be json, but it keeps the same structure.

The site I am making these pages for is an .net MVC site. I'm using razor for everything else, but I think for this particular page there may be a better approach.

The concern I have isn't in parsing of the data. I know you can use dynamic objects with many libraries. The concern is formatting the data once I have it. There are no good identifiers or tokens to use to step through data formatting it. With these dynamic objects is there a good way to format them before passing the model to a view, or Am I going to have to write a huge xsl sheet for handling all possible cases. There are 2000+ different fields that items can have and more could be added so I don't want to do it that way.

Formatting is an issue because they want stuff like authors to be formatted differently based on category APA vs MLA stuff like that as well date formats and phone/ mail formats to differ based on category. The problem orginates from the APIs which I can't fix now, not my project nor are they listening to me. When the the data structure for the backend of the APIs was set up they wanted to allow for any data which is good, but they didn't set up and structured guidelines for formatting or creating new fields.

So publication may have authors while poems may have author and articles may have AUTHORS. Date in one category my be date or startdate or enddate or birthday. I realize there isn't a very elegant solution to this without fixing the root cause was just hoping someone had some advice for a quick easy interim solution until the APIs can be tackled right. Also they want me to reorder fields in some spots not based on category but based on adjacent fields.

Answer 1

Your going to have to write a lot of custom formatters. As a example, he's some solutions for formatting phone numbers:

https://stackoverflow.com/questions/188510/how-to-format-a-string-as-a-telephone-number-in-c-sharp

As you can see, lots of variations on a theme.

You might try a factory approach where where you pass a type and a category and it returns a formatter. All formatters inherit from an interface like IFormat. Some examples:

var formatter = FormatFactory.Create(FormatType.Telephone, "CategoryX");
var formattedString = formatter.Format(phoneNumberFromJsonString);

var formatter = FormatFactory.Create(FormatType.Author, "APA");
var formattedString = formatter.Format(AuthorFromJsonString);

At least you would be able to keep the format logic focused instead of having a single formatter trying to handle all the scenarios.

Answer 2

Excusme in advance for retyping the answer. As @WinstonEwert pointed I ignored the main problem of the question.

Summarizing the comments:

• Data model (EAV) has not been not normalized.
• Data formats have not been applied to the data model.
• In consequence, XML/JSON are not normalized either. So, the data remains unformatted after the serialization.

It's a sequence of problems that can not be addressed at the end of the chain because we would be facing them altogether.

The solutions exposed here: formatters, XSL, regular expressions, etc. Seems palliative solutions. They might stand for a little while, but they will become insufficient eventually.

The real problem is in the data model. As developer, you have done what is expected from developers to do. To notify the problem and the risks. Now is not in your hands.

Said this, if there are no resources (or will) to address the weakness of the system, I would suggest first a management of expectations.

With unlimited resources (time and money) everything is possible. But I doubt your company has that much of them. So they will have to be flexible at some point.

Measure the costs of the exposed approaches, explain how the technical debt is progressively leading the company to an endless work (due to new elements can be added to the system) with no ROI in the near future. The actual system seems inefficient and near to impossible to automate or to programme the task so that the maintenance becomes affordable.

Once everybody has been warned about the risks and agreed, you can start to code.

Answer 3

It sounds like you can't rely on structure. Instead you will have to rely on values. I suggest you write something akin to a command line processor that tailors its behavior to each of the keywords.

Steps:

1. Take inventory of all the element keywords in the database. Perhaps you could write a simple program to iterate through the data while capturing Fiddler logs. Throw the logs into a word processor and take a word count.

2. Identify all the words that require special handling, Hopefully there aren't too many.

3. In your program, assemble a Dictionary<String,Func<string, string>>, with one entry per special keyword. The key in the dictionary is the keyword: the value is an anonymous function that accepts a string and returns a string.

4. Populate the dictionary with functions that are capable of performing the special handling per keyword.

5. To format a page, retrieve the XML and iterate through its elements using XPath or Linq-Xml. For each element, look up the anonymous function in the dictionary and call it, passing the contents of the XML node.

6. Retrieve the result of the anonymous function and put it on the page.