6

I don't know whether to call the data dynamic or inconsistent. But I need to create profile pages for people generated from xml or json. The challenge is in the data the is returned. The data is bibliographical data about a person, and the APIs that are returning it are old, 11 years old. They need revisiting.

the data can look something like this

    <person>
         <personalinfo>
              //Always the same......
         </personalinfo>
         <categories>
             <publications>
                 <item> //<---The issues lie inside here.
                      <authors>
                      </authors>
                      <publications>
                      </publications>
                 </item>
             </publications>
         </categories>
    </person>

The issue lies with the categories and item information. A new category can be added at any time and be called anything. Also an item can have any fields and those fields can have any name and be added at any time. Essentially I don't know what I am getting back. On top of that there is no token returned giving any hints about displaying the information nor is there any format requirements on data that is entered into the feed that these APIs are returning.

I know that these APIs need updating but that isn't on the table right now. I just got a deadline pushed forward 2 weeks and need to have profile pages done very soon.

Are there any good tools that can handle this mess of information? Does anyone have any suggestions for getting this done quickly. This is going to be an iterative project I imagine so whatever I use is most likely going to be an interim solution. The data returned can be json, but it keeps the same structure.

The site I am making these pages for is an .net MVC site. I'm using razor for everything else, but I think for this particular page there may be a better approach.

The concern I have isn't in parsing of the data. I know you can use dynamic objects with many libraries. The concern is formatting the data once I have it. There are no good identifiers or tokens to use to step through data formatting it. With these dynamic objects is there a good way to format them before passing the model to a view, or Am I going to have to write a huge xsl sheet for handling all possible cases. There are 2000+ different fields that items can have and more could be added so I don't want to do it that way.

Formatting is an issue because they want stuff like authors to be formatted differently based on category APA vs MLA stuff like that as well date formats and phone/ mail formats to differ based on category. The problem orginates from the APIs which I can't fix now, not my project nor are they listening to me. When the the data structure for the backend of the APIs was set up they wanted to allow for any data which is good, but they didn't set up and structured guidelines for formatting or creating new fields.

So publication may have authors while poems may have author and articles may have AUTHORS. Date in one category my be date or startdate or enddate or birthday. I realize there isn't a very elegant solution to this without fixing the root cause was just hoping someone had some advice for a quick easy interim solution until the APIs can be tackled right. Also they want me to reorder fields in some spots not based on category but based on adjacent fields.

  • Newtonsoft.Json will return a dynamic object, which you can then introspect with Reflection. HTML Agility Pack should already do this for you for the XML data. – Robert Harvey Oct 14 '16 at 15:01
  • What requirements have you been given in terms of how to format this? As its stands, your question is unanswerable, because we have no idea what your requirements are in terms of how you are supposed to present the data. Should you show a table? Give the user the raw xml? Interpret the xml as raw image data and show it as an image? Are you supposed to present customized widgets for each category? Can you take a hash of the XML and show that? Without better requirements, there is no way to answer this question. – Winston Ewert Oct 14 '16 at 18:22
  • They want a page for viewing a person bibliographical data, they want to be able to look at different categories at a time. This part is easy. The hard part is they want to fields data to be formatted based on stuff like what category it is in and what are the adjacent fields. I told them this seems unrealistic and formatting of the data in the fields should have been handled before data was entered or by the API; somehow or another I am left responsible for handling it. EDIT:unrealistic because of the sheer number of combination some of it could be handle programatically,but 2000+ field types – PieMaker Oct 14 '16 at 20:36
  • Can you give an example of a case where an adjacent field changes the formatting? – Winston Ewert Oct 14 '16 at 20:39
  • easiest example is start date and end date wants to be concatenated... There are quite a few that are a lot more involved. Most of it involves concatenating data and slight formatting when other fields are present. – PieMaker Oct 14 '16 at 20:40
1

Your going to have to write a lot of custom formatters. As a example, he's some solutions for formatting phone numbers:

https://stackoverflow.com/questions/188510/how-to-format-a-string-as-a-telephone-number-in-c-sharp

As you can see, lots of variations on a theme.

You might try a factory approach where where you pass a type and a category and it returns a formatter. All formatters inherit from an interface like IFormat. Some examples:

var formatter = FormatFactory.Create(FormatType.Telephone, "CategoryX");
var formattedString = formatter.Format(phoneNumberFromJsonString);

var formatter = FormatFactory.Create(FormatType.Author, "APA");
var formattedString = formatter.Format(AuthorFromJsonString);

At least you would be able to keep the format logic focused instead of having a single formatter trying to handle all the scenarios.

  • I figured I would have to do something like that eventually. Was hoping there was a faster interim solution that might still look fairly decent. Honestly I feel most of this formatting of data should have been handled at different points along its journey not all right before it needs to be displayed. I may just start working towards this and use javascript and a small amount of formatting code as an interim solution. Thank you for the input. – PieMaker Oct 14 '16 at 21:05
2

Excusme in advance for retyping the answer. As @WinstonEwert pointed I ignored the main problem of the question.

Summarizing the comments:

  • Data model (EAV) has not been not normalized.
  • Data formats have not been applied to the data model.
  • In consequence, XML/JSON are not normalized either. So, the data remains unformatted after the serialization.

It's a sequence of problems that can not be addressed at the end of the chain because we would be facing them altogether.

The solutions exposed here: formatters, XSL, regular expressions, etc. Seems palliative solutions. They might stand for a little while, but they will become insufficient eventually.

The real problem is in the data model. As developer, you have done what is expected from developers to do. To notify the problem and the risks. Now is not in your hands.

Said this, if there are no resources (or will) to address the weakness of the system, I would suggest first a management of expectations.

With unlimited resources (time and money) everything is possible. But I doubt your company has that much of them. So they will have to be flexible at some point.

Measure the costs of the exposed approaches, explain how the technical debt is progressively leading the company to an endless work (due to new elements can be added to the system) with no ROI in the near future. The actual system seems inefficient and near to impossible to automate or to programme the task so that the maintenance becomes affordable.

Once everybody has been warned about the risks and agreed, you can start to code.

  • A downvote well worth a Why? – Laiv Oct 14 '16 at 20:28
  • 1
    -1 for ignoring the question's explicit statement that it was about formatting the data not parsing. – Winston Ewert Oct 14 '16 at 20:29
  • True. I miss read the Isn't a parsing. My bad. – Laiv Oct 14 '16 at 20:32
1

It sounds like you can't rely on structure. Instead you will have to rely on values. I suggest you write something akin to a command line processor that tailors its behavior to each of the keywords.

Steps:

  1. Take inventory of all the element keywords in the database. Perhaps you could write a simple program to iterate through the data while capturing Fiddler logs. Throw the logs into a word processor and take a word count.

  2. Identify all the words that require special handling, Hopefully there aren't too many.

  3. In your program, assemble a Dictionary<String,Func<string, string>>, with one entry per special keyword. The key in the dictionary is the keyword: the value is an anonymous function that accepts a string and returns a string.

  4. Populate the dictionary with functions that are capable of performing the special handling per keyword.

  5. To format a page, retrieve the XML and iterate through its elements using XPath or Linq-Xml. For each element, look up the anonymous function in the dictionary and call it, passing the contents of the XML node.

  6. Retrieve the result of the anonymous function and put it on the page.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.