c# xml import and validation with different document versions without XSD

Question

Context:

I have a case where I need to extend some existing functionality for loading a new version of an xml file into a system (new fields, some mandatory that were not mandatory before, etc). The file contains some heavy information, but to keep things simple I will use a simple case.

The xml file can be something like this.

Version 1 (used in current implementation):

<?xml version="1.0" encoding="utf-8"?>
<version value="1" />
<items>
  <item name="item1">
    <subitem name="subitem1" value="value1"/>
    <subitem name="subitem2"/>
  </item>
</items>

Version 2 (new version, not implemented yet):

<?xml version="1.0" encoding="utf-8"?>
<version value="2" />
<items>
  <item name="item1">
    <subitem name="subitem1" value="value1" description="sub item 1"/>
    <subitem name="subitem2" value="value2"/>
  </item>
</items>

• In Version 1 the value property in subitem is not mandatory.
• In Version 2 the value property in subitem is mandatory, and there is a new optional property called description.

Current Implementation:

Currently, the system was developed with hard coded classes (because of some complexity during class matching between the xml and the db), and for the mandatory value validation, a custom attribute was used.

Something like this (only import/export of XML Version 1 is supported):

public class Item
{
  [Mandatory]
  public string Name {get;set;}

  public IList<SubItem> SubItems {get;set;}
}

public class SubItem
{
  [Mandatory]
  public string Name {get;set;}

  public string Value {get;set;}
}

This was also developed like this, because at some point during the validation process, the missing properties will be shown to the user so the user can fix them right away (by inserting the value in a TextBox for example).

The problem of having this structure is that a new version that needs to change which properties are Mandatory or not, requires changing the current implementation, and of course would not work for previous versions.

There are currently no XSD schemas for either version, just a document defining this new version (which fields are mandatory, which ones are new, etc).

Possible approaches:

One option I have is to change the class definition to be abstract, and override the properties that changed in a new version, but if there are multiple new versions it will become hard to read and hard to follow (multiple versions, with multiple properties changed, and multiple validations changed for the property), so I'm not sure this is a good approach.

Another approach is to remove the custom attributes, and add another layer of validation using something different (but I'm not entirely certain as to what to use). Possibly using a strategy pattern with methods for validation depending on the version.

Question:

How else can this problem be addressed? Is there a design pattern that I can follow for this type of scenarios?

Your help is much appreciated.

Answer 1

The problem with different versions of an XML document is that not every change is compatible with previous versions. Coding around that can be difficult. There's a few approaches you can take:

• Have a separate set of DTO objects depending on the version
  • Use XQuery to read the version attribute so you know which document root to serialize to
  • Major disadvantage is that most likely means you have an application model you need to populate so the rest of your API is OK
• Generate the XSD from the textual document
  • Validation is part of reading the XML document, so it doesn't have to be baked in to your DTO objects
  • You have to manually update the XSDs and keep one for each version in your code
  • Use XQuery to determine which XSD you need to apply since the namespacing won't resolve that for you
• Separate validation from serialization issues
  • Your DTOs don't have any validation directives whatsoever
  • You write validators that apply the validation rules against your DTOs
• Use XSLT to conform the different versions to whatever the current version is
  • Only possible if the changes to the XML are predictable and resolvable
  • Advantage is there is only one DTO set of objects to manage
  • Use XQuery to determine the version so you can apply the right transform
  • XSLT can also transform values so if an enum changes to an URI you can map values to what you need to key off of in your processing code
• Have completely different microservices for each version of the schema so that you can have multiple versions running simultaneously if you have a transition period.
  • All code is verified and debugged for the version of the schema you have to deal with
  • You can retire the old microservice when you no longer receive messages of that version.
  • Either external code hits the correct version of the microservice directly, or you have a proxy or redirect that forwards based on the results of the XQuery operation to get the version.

Because it's a messy problem, you have to make the choices depending on how different the XML "schema" are, how flexible your code needs to be, and whether there is a long transition period.

There's a number of subtle problems that you really need to be aware of when you have versioned XML documents like this:

• Items that used to be mandatory that are now optional: You will likely need to update your processing code to deal with the possibly null values. Assumptions that used to be valid are no longer valid.
• Items that used to be optional that are now mandatory: Your processing code is probably OK, but you might have more rejected XML documentation if they have the wrong version number
• Added XML elements/attributes: Either ignore the information, or add processing code that does not assume the new XML elements are present--even if they are marked as mandatory (remember multiple versions).
• Removed XML elements/attributes: Your processing code needs to treat those values as optional, or perhaps calculate replacement values for data consistency.
• New enum values, or changes to data formats: This is the messiest to deal with, particularly if you are keying off of a couple values to determine if a record gets set.