Over the past year or two, I've been playing with newer technologies in my side projects. As a web developer, I've gone from the following (and still the following, at work):

The 'classic' technology stack

  1. Web browser POSTs forms to...
  2. a C#-coded web application server-side, communicating to...
  3. external services via XML, then ultimately...
  4. writing the application state to a SQL database.

To using a very different arrangement:

The fully-dynamic technology stack

  1. Web browser submitting XmlHttpRequests to...
  2. a JavaScript-coded Node.js server-side, communicating to...
  3. other external services via RESTful services in JSON, ultimately...
  4. writing the application state to a no-SQL database

It's gotten to the point where my whole stack has absolutely no enforcement of type or schema, anywhere.

Now, this has been just fine before, when consuming others' web services. It might have even been fine up until the SQL databases were ditched (along with their DB schemas). But now I'm stuck at:

Where do I declaratively define what is the valid structure of my business-domain data?

I want to enforce data validity before making any of my projects publically available - after all, what's to stop someone from just submitting invalid data to my services, and using it all as just a hacky free database provider? At some point, it has to be enforced that "this web service will only accept a collection of at least one X. X must contain A, B, and optionally C".

The first solution I thought of was to do validation inside my node.js, through a big block of imperative code/if-elses/etc. This felt wrong.

I have been using CouchDB, and for a while I thought that it might be best to put that validation code in the _update handlers. At least we're performing validation as part of persistence, but it's still an ugly imperative block.

Next, I looked into JSON schema languages. There is no standard as far as I can see, and I wasn't confident in the multiple solutions offered. I could roll my own, but then I'd only be expanding the body of non-standard JSON schema languages.

XML? I could put the X back into AJAX, and have them all schema-validated on the server side. That doesn't seem to follow the trends in software development, however. Neither would using an XML data store instead of a JSON CouchDB/BSON MongoDB persistence layer.

So, I'm stuck. Ideas?

TL;DR Where do I declaratively define the valid structure of my business-domain data when I am using no static Object Oriented language, and no schema-bound SQL database, in my technology stack? Is it possible that there must be some static typing (or DB schema, or validated XML) somewhere for a technology stack to make any sense? If so, where?

  • I guess I should make it clear that yes, I understand that flexibility of structure is a main advantage of JSON-based databases, and also JSON-based data encoding broadly. Though it should still be important that, given the current time/context, you're at least receiving and storing correct data, yes? – Stoive May 10 '12 at 5:15
  • Now I understand that in trying to express this question, I've muddled the concepts of 'data type' and 'data schema'. The question is about schemas. static OO languages (Java, C#) impose a schema via your class definitions of business types. SQL can also provide this via its DB schema. But a fully-dynamic, no-SQL stack has neither of these, leaving validation to in-code checks, which I guess I've lived without before. – Stoive May 10 '12 at 7:09
  • 1
    IMHO "dynamic" should be called "untyped" ...and the few keystrokes you spare by not having to type these have their toll when the code grows. Nothing is checked for you, you have to do it by yourself, in an imperative way, as well as test things extensively. It's intrinsic to the "untyped" paradigm. My 2 cents. It's also interesting to see that back in the ages, VB was bashed and blamed because it contained dynamic types, while now they are raised by hype. At least VB had both, typed and untyped variables. But it looks like IT is more driven by hype than reason. (putting flame-proof suite on) – dagnelies May 10 '12 at 8:49
  • That is to be expected when using poorly typed languages. The whole point of using computers is the manipulation of data structures. Instead of taking care that these data structures are well-defined they are intentionally ignored. To add insult to injury this is proclaimed by some as the way forward. – ThomasX May 10 '12 at 11:15

Sounds to me like you're about to create an inner platform to compensate the lack of semantics inherent to the language you chose.

Even schema definitions (be it XML DTD, or JSON schema or mongoose schema) will not provide you the safety of statical analysis. All you can really use them for is to guarantee that your system doesn't silently run into undefined behavior and ultimately fail.

I am not really sure why you won't use a language, that simply provides this out of the box. Using a dynamically typed language and embedding type constraints into that seems to combine the worst of both worlds. Even more so, because modern statically typed languages are able to infer vast parts of those constraints implicitly.

Personally I suggest you take a look at Haxe's JavaScript backend. There's a site dedicated to node.js development with Haxe - you could start there. Haxe's anonymous types can quickly be used to tie in JSON sources in a type safe manner without any runtime overhead. Still, if you wish, its meta programming facilities allow you to automagically generate validation code from that at compile time.

Of course Haxe is by far not the only option out there to target JavaScript in a typesafe manner. So if you feel you need the benefits of static typing, the you should invest the time to find a language to your liking, that actually embeds this information right into its statically analyzable semantics.

| improve this answer | |
  • 3
    Static typing != input validation. – Joeri Sebrechts May 10 '12 at 11:53
  • 2
    @JoeriSebrechts: Any decent (i.e. sufficiently reflective) statically typed environment offers the possibility to generate basic validation from the type information (and some sort of annotations if necessary). Take: typedef Person = { firstName:String, lastName:String, email:Email, phone:Phone, mobile:Null<Phone> }. This has enough information to validate a registration form. I am not saying that static type systems can express all relationships. But it can express all the relationships one would find in a schema definition. – back2dos May 10 '12 at 12:49
  • 1
    @JoeriSebrechts: I disagree. Decent type systems allow to formulate these kinds of constraints. In fact pretty much any kind of constraint that checks the validity of a field in isolation. From the above type definition, you can easily generate validation code, that will check whether the provided phone number is valid for example. Why should I write any kind of code to do all this by hand, if I can embed this information into my language and then use metaprogramming to generate the needed code with it? – back2dos May 11 '12 at 9:00
  • 1
    @JoeriSebrechts: Take a look at Haskell's type system... – tdammers May 11 '12 at 12:33
  • 1
    Input validation is sometimes called the "boundary issue". When dealing with the boundary issue, static types is not enough, because you will never have enough information to statically elide all type information at compile time. You are forced to dynamically wrap and unwrap and check data just like how dynamic types work in dynamically typed languages, but now this only occurs at the boundary. This means we either have to embrace a dynamic typing methodology similar to Clojure's schema (github.com/plumatic/schema) or embrace dependent types. Once data is validated it can be tagged. – CMCDragonkai Sep 12 '16 at 11:36

Ditch the notion of types altogether.

The whole point of dynamic typing, at least in languages that get it mostly right, is that you don't have to think about types most of the time. Type-checks make sense in a pre-compiled language, because the compiler can then catch a lot of errors before you can even attempt to run the code, but when your code is interpreted or JIT-compiled, the benefits of static type checks are marginal and don't weigh up against the added flexibility that dynamic programming has to offer.

Instead of validating types, validate values - that is, if you want something to be numeric, run an 'is-numeric' check. If you want something to have a certain property, check if the property exists. If you want something to be in a certain range, check if it is in fact within the range. Etc. etc.

There are two philosophies to this: the more traditional check-before-you-go, that is, before dividing a by b, you make sure b is non-zero; and the easier-to-ask-forgiveness-than-permission route Python takes, that is, you just divide a by b, and then catch the divide-by-zero exception afterwards.

Unfortunately, there is no meaningful way of doing this declaratively that I am aware of. But then, the boundary between code and data is much blurrier in dynamic programming - code is also data that you can manipulate, and data can take the role of code. A piece of javascript code that validates a JSON object is just another chunk of text, and you can treat it as plain old data; you can even generate that text at run-time and then execute it (although you should be very careful about this, especially if you use data from untrusted sources in the code generation process).

As far as schemas go, again, checking strictly whether a given JSON object matches a schema is seldom required - instead, you just check whether the required properties are there, and whether your value-level invariants hold. If extra properties exist, that's fine - you just ignore them. If something is missing, you can either bail, or substitute a default value. And if something has the wrong type, again, you can bail (found a non-numeric string where a number was expected), or you can gracefully recover (substitute a default value, round, truncate, convert to boolean, etc.).

Another thing to consider is that XML is far more complex than JSON - XML has namespaces, attributes, entities, and you need to decide how to marshal between objects in your language and XML representations. In JSON, the choice is usually obvious - it's either scalar (int, float, string, boolean, null), a simple list (array), or a key-value collection (object).

Unfortunately, there are few languages (if any) that get transparent type casting right - Javascript, PHP, Python, they all require explicit type juggling occasionally, so you won't be able to reach the theoretically ideal situation.

| improve this answer | |
  • I was getting ready for bed and I run into Deep Programming Philosophy®. Now I'm not going to be able to get to sleep for an hour. Thanks! :-) – Peter Rowell May 10 '12 at 6:09
  • 1
    +1. Although I don't share your belief on dynamic typing being more flexible and static typing having only marginal benefits ;) – back2dos May 10 '12 at 6:10
  • 1
    I have no problem at all with writing in dynamically typed languages, and I'm definitely not keen on imposing type systems on them. The problem is that "check before you go, only for what you need, ignore the rest" by nature tends to be clunky, imperative code. At least SQL-backed databases/static OO languages imposed a 'schema' (in the relational/OO realms, respectively), but nothing in the newer stack described above intrinsically contain any schema validation. – Stoive May 10 '12 at 6:52
  • 1
    The whole point of dynamic typing, at least in languages that get it mostly right, is that you don't have to think about types most of the time. The whole point of using computers is the manipulation of data structures. So you are saying "I want to manipulate data structures without thinking about those data structures". – ThomasX May 10 '12 at 11:18
  • 1
    @tdammers I've seen some pretty large government data exchange projects here in the Netherlands where SOAP is extensively used, applying XML schema's totaling 10's of MB large. Yes, it is complicated, but it would be near impossible to prove any correctness in sth such as JSON. Writing validation for these structures would be near impossible without reinventing the wheel and creating a data validation language. I'm not stating XML schema or XML is readable, it is not. You should use graphical editors to help you interpret. – Dibbeke May 11 '12 at 11:24

Well as your programming stack does not care about types why should you.

The answer is that your users might. So what you want to do is enforce whatever business rules apply to your data.

This can be done by JavaScript on entered forms, by coding validation into your Node.js server code when a request is received, or, even encoding the rules in xsd and getting the parser to validate your xml input.

The point is you implement business rules validation not technical validation. That is you should be checking rules like "A three digit number between 001 and 700" which mean something to your end user, rather than "is this numeric". So if your user thinks "The night before christmas" is a valid date and your application can live with it then its a valid date.

| improve this answer | |
  • Ahh, I think you've understood what I'm trying to ask! Enforcing business rules - an SQL database enforces some of them through the schema, and a strongly-typed OO language helps enforce them through all the classes you have to create. Guess I just need to be less lazy, and just write some lines in my code to do this :p – Stoive May 10 '12 at 7:34
  • 3
    Always a bad idea to let the DB validate your data anyway. Makes error handling really messy! – James Anderson May 10 '12 at 8:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.