Why validate XML?

Question

I am looking for an explanation on just why XML needs to be validated. I have been testing DTD's and Schema's for the past month and recently tried to find out why I am doing this. I know it makes the XML better and more Semantic but what are the overall benefits of validating XML?

Answer 1

To make sure that it works. Sure, if you only write out some data for your own application to read it may be enough if it just works. If you send a file to somebody else matters may be different. But even within your application you may later choose to switch the parsing library and the new one may complain about errors the old one accepted and ignored like not properly escaping some characters.

Or, depending on the kind of data you store, some new characters may appear (maybe a user with a foreign name) and from that moment proper character set definitions may be important. Same if you just send an XML file from Windows to a Linux system. The closer you keep to definitions and validations the less likely you run in unexpected trouble when things change.

Answer 2

It depends on how you're using the XML.

For example if you wish to pick out one or two pieces of data out the XML then move on then it would be best just to extract them and move on.

However if you need the whole XML document, so for example you want to transform it to some other XML format that needs all the fields, then I would get a schema involved.

It also depends on the relationship you have with the person sending you the XML : if you're receiving XML from a party you need to be formal with (e.g. another business) then having XML schemas allows you to establish a contract about what you will accept as input and give as output.

On the other hand if you have an informal relationship with then you can be generous with what you will accept / reject.

Answer 3

It's a bit like asking, why do we test our programs. We do it in case they are wrong.

Of course there are scenarios where you know the data isn't wrong, because the program that generates it is well tested and well trusted. In that case you don't need to validate the data. It's not compulsory.

Answer 4

Well, you don't need to. Validating XML against a schema (DTD is outdated garbage) is equivalent to using a statically typed language. It ensures a higher level of correctness at the expense of more upfront labor. I generally find that validating XML is very useful when you're dealing with remote services. It's self-documenting. If you're just transferring data from a client to server where you control both sides, then it's less useful.

Answer 5

Speaking in terms of using XML for web services: A well-formed schema lets the consumer of the web service know what the server expects and then what the server will give you back. It really helps knowing what the inputs and outputs for the web service are.

Then there are the badly-formed web services: they tell you what they expect from you, but then return some nonsense. I am currently dealing with such a service: they tell you how to form the input, but then they return the response as a string. A string containing complex XML. Said complex XML has no schema to validate against, so I have no idea what I am guaranteed to get back from the server, what is optional, or what types are possibly returned. It is effectively useless, as it tells me nothing useful.

Answer 6

I think the question could be phrased like this:

When programming a function, should I check my function arguments to make sure they are not null and of the right type and perhaps within the correct range?

Why is this the same question? Basically one validates "input" whether it's XML or a simple object. Since I have vastly more experience in writing code than making XSDs and validating XML, the re-phrased question seems much simpler to answer.

Not that it is simple, just simpler. Overall I try to avoid validation of function arguments - there's no end to how much validation code I could have. And if I code both caller and callee I should know which arguments is sent so no need to validate anything. So - I think my first rule would be: Do not validate - make sure to write robust code in the first place. Sometimes you will get a NullPointerException - but your application must be prepared to handle that. Log, investigate and improve your code. Remember that if something fails, validation couldn't really fix anything - it's mostly a matter of where the error is detected.

However, if the input is received from a human or a very advanced system then validation will make sense. The assumption will then be that the input-producer is able to process an error message from the validation and correct the input. This is the perfect case for validation because it actually prevents errors from reaching your function.

I find that I sometimes still do validation in my function for clarity of code or readability, it can be hard to read from code what an input object might contain. Mostly the checks will be fairly simple (is it null or not?)

Sometimes validation is a sign of distrust of the caller (I haven't made it myself, so anything could happen - instead of investigating a little). This is a sign of laziness. In such cases, validation can obscure the data by introducing a false notion of uncertainty. In turn, it will make refactoring harder.

You might have a function which acts as a filter - only data with a certain quality is allowed to pass. Then validation will make sense.

I am sure there are much more to say about this - but thinking along these lines, to avoid adding anything unnecessary to your system/code, is hopefully useful.

Answer 7

Why? because you don't want to write validators yourself, and if you do, it is very likely you aren't experienced in doing so, and when you need to change it to meet new requirements, you'll have an awful time.

Despite common belief, XML is not rigid; as long as your XML is syntactically correct, it is valid XML, despite the ignorant people telling you otherwise.

You WANT to validate your XML if your applications(be it web or otherwise) rely on models(not necessarily XML, because any model can be transformed to XML by writing a simple tree walking program) to behave correctly.

For example, you might have a JSON model for colors

[
    {"name": "red", "cssxrgb": "#ff0000"},
    {"name": "blue", "cssxrgb": "#00ff00"},
    {"name": "green", "cssxrgb": "#0000ff"}
]

Most programmers, if asked to write a program to make sure that all entries in this array are valid, and say what isn't valid when it isn't, will write something of this smell:

var jsonString = `
[
    {"name": "red", "cssxrgb": "#ff0000"},
    {"name": "blue", "cssxrgb": "#00ff00"},
    {},
    {"name": "green", "cssxrgb": "#0000ff"}
]
`;
var data = JSON.parse(jsonString);

for (var i = 0; i < data.length; ++i) {
    var row = data[i];

    if (!row.hasOwnProperty('name')) {
        throw new TypeError('row #' + i + " is missing required property name");
    }
    if (!row.hasOwnProperty('cssxrgb')) {
        throw new TypeError('row #' + i + " is missing required property cssxrgb");
    }
}

This can be difficult to maintain, requires very careful coding, and is very fragile to typos, field changes, so on.

A simple program can be written to transform the above to XML, and most models already have many online transformers from them to XML. It doesn't even have to be a good transformer, It's easier to change the method of transforming data to xml than changing custom validators. Here's an example for transforming above to

<colors><color name="red" cssxrgb="#ff0000" /><color name="blue" cssxrgb="#00ff00" /><color name="green" cssxrgb="#0000ff" /></colors>

transformation code:

var jsonString = `
[
    {"name": "red", "cssxrgb": "#ff0000"},
    {"name": "blue", "cssxrgb": "#00ff00"},
    {"name": "green", "cssxrgb": "#0000ff"}
]
`;

var data = JSON.parse(jsonString);
var xmlString = '<colors>';

for (var i = 0; i < data.length; ++i) {
    var row = data[i];
    var name = row.name;
    var cssxrgb = row.cssxrgb;
    var keys = Object.keys(row);
    xmlString += '<color ';    
    var tmp = [];
    for (var j = 0; j < keys.length; ++j) {      
        var key = keys[j];
        var value = row[key];
        tmp.push(`${key}="${value}"`);
    }    
    xmlString += tmp.join(', ') + ' />';    
}
xmlString += '</colors>';
console.log(xmlString);

Now to validate, all you have to do is run it against the doctype such as

<!ELEMENT colors (color*)>
<!ELEMENT color EMPTY>
<!ATTLIST color name CDATA #REQUIRED>
<!ATTLIST color cssxrgb CDATA #REQUIRED>

You can check it with online validators as such

<!DOCTYPE colors [
    <!ELEMENT colors (color*)>
    <!ELEMENT color EMPTY>
    <!ATTLIST color name CDATA #REQUIRED>
    <!ATTLIST color cssxrgb CDATA #REQUIRED>
]>
<colors><color name="red" cssxrgb="#ff0000" /><color name="blue" cssxrgb="#00ff00" /><color name="green" cssxrgb="#0000ff" /></colors>

The strict validators will reject the model if it contains any attributes/elements that aren't declared in the DTD, which is useful for when your programs are fragile to key,value pairs that it doesn't expect. The problem is if a validator stops validating after first error, or if there are too many false positives like undeclared tags than violations of the things your doctype actually asks to check for.

If you do know how to write your own validators, you can often write better ones faster than writing your own model-to-xml transformers and your own doctype, but even in this case, it's easier to change an existing DTD or model to xml transformer than having to modify your own custom validator.

As for HTML/SVG/other common formats; They are usually invalid XML, but if you do choose to make them valid XML, you get the ability to add your own tags which are ignored by the document if you style them to display: none, but still have feedback when you accidentally made a typo in your own model. This is useful for things like angular templates, custom metadata that gets handled by JavaScript when certain events trigger.

In the very worst, it lets you have a way to tell which parts of DOM are "standard", and which are either metadata(eg author tag in head)/custom(eg gallery tag that angularjs will transform later)/junk data(typos, metadata that's no longer needed, so on), in each case, those custom tags introduce complexity to your model, and validating xml can help you be aware of which parts of DOM might be causing the most trouble.

To summarize here are the benefits of validation

1. it is validatable - even if it doesn't use validation, you can at least know that it is possible to validate because it has a syntactically correct XML structure, allowing you to validate it when you want to or when need comes.

2. It helps find parts of DOM that a program depending on the model might not expect.

3. It is simpler to write and maintain a simple validator by transforming to XML and writing a DOCTYPE and pick and choose which DOCTYPE to use on a specific validation run than writing your own validator. It is also simpler to fix edge cases in your custom transformer, doctype than inside a custom validator.

4. It can exploit XML technologies - if it's valid XML, it can use many of the XML technologies such as XSL easily. These technologies usually add a lot of complexity over doing things conventionally, but they do allow pretty powerful ways to control your document once you get comfortable with any of them.