Can a system be 100% Data Driven?

Question

My new boss has been working on this project for many years. I've only been here a few weeks, but I am not sure it's possible. He would like to design a system that is "100% data driven".

So if we put in enough data, we can define and generate any application. I've managed to at least get him to concede some things like users, or apps should have predefined values, but he likes the concept of the structure of the system, the user interface and the logic all being stored as data.

There are some demos of simple things and he's basically rediscovered some simple ideas of object oriented programming and your basic template systems, but I think overall that this goal might be actually impossible.

I don't know how you can define logic using data without the system becoming so complex that you are doing actual programming anyway.

I think theoretically it isn't because the thing that interprets the data ends up needing to become turing complete to describe the application so you've just shifted the problem one level higher to no net benefit.

Is such a 100% Data Driven Application possible?

Answer 1

Your boss should read this piece: Bad Carma: The "Vision" project, a cautionary tale about inner platform effect or second system effect.

Abstract

Those of us who work in Information Technology (IT) have all been on a project where something important is just not right. We know it, most everyone knows it, but nobody is quite able to put his or her finger on the problem in a convincing way.

This story is about such an IT project, the most spectacular failure I have ever experienced. It resulted in the complete dismissal of a medium-sized IT department, and eventually led to the destruction of a growing company in a growing industry. The company, which we'll call "Upstart," was a successful and profitable subscription television business.

The project occurred in the early 1990s, and it was a custom-built order-entry and customer-service application, closely resembling what is now referred to as Customer-Relationship Management or CRM. The core functionality of the system included:

• Order entry and inventory
• Customer service, help desk
• General ledger, accounts receivable, billing, and accounts payable

The application was called "Vision" and the name was both its officially stated promise for Upstart as well as a self-aggrandizing nod to its architect. The application was innovative, in that it was built to be flexible enough to accommodate any future changes to the business. Not just any foreseeable future changes to the business, but absolutely any changes to the business, in any form. It was quite a remarkable claim, but Vision was intended to be the last application ever built. It achieved this utter flexibility by being completely data-driven, providing limitless abstraction, and using object-oriented programming techniques that were cutting-edge at the time.

Like many such projects that set out to create a mission-critical application, the development effort spanned two years, about a year longer than originally projected. But that was acceptable, because this was the application that would last forever, adapting to any future requirements, providing unlimited Return On Investment (ROI). When the application finally went "live," almost everybody in the company had invested so much in it that literally the fate of the company hinged on its success.

However, in the event of total project malfunction, mission-critical applications running the core business of multinational corporations are not permitted the luxury of the type of fast flameout demonstrated by thousands of "dot-com" companies in the era of the Internet bubble. Within a month of Vision going "live," it was apparent to all but those most heavily vested in its construction that it was a failure.

See Also

http://en.wikipedia.org/wiki/Inner-platform_effect

Answer 2

The answer is yes, it is possible to create a fully data-driven system and yes, it's usually a really bad idea.

A fully data driven program is one in which all logic and configuration is handled by values stored in such a way that in another context they would be regarded as data. There were many 4GL products produced in the 1980s that provided the capability to generate reports, forms, tables and logic using data items entered into a multiplicity of forms, stored in tables and accessible through reports. I used to refer to such systems as "paint by numbers" but I see it has now come to be known as the "inner system" effect. Good name.

People who create these systems are trying (whether they know it or not) to create a new programming language. Since they do not have the skills, they do it badly. From the viewpoint of the JVM/CLR, a compiled Java/C# program is simply data. In this case it has been done well. In either case, programmers are needed to use the language, whatever it is.

There is one specific way to make this work, that I know of. You build the skeleton of each of the components you need: form, report, table, etc. You provide a mechanism to configure various parts of these components by setting data items. For a chosen set of features you make the decisions and freeze them into the system, and specifically deny the ability to configure those features.

You also implement a language that has the ability to code logical operations. My recommendation is to use an existing language such as lua or maybe Python. You embed pieces of this code wherever logical operations are needed.

By doing this you substantially reduce the amount of writing required to implement each form, report, table and so on. The system seems to be data-driven, but only to a point.

At this point you have just implemented a new 4GL. If you happen do do this successfully, please let me know. Most people fail dismally. I will be the first to congratulate you on your achievement.

Answer 3

I think you're basically correct. A language runtime is already a fully flexible data-driven system. It takes one piece of data (the program) and uses it to determine how it should act on other data. It might even have a multi-user scheme to store code for re-use by other programs (ranging from an include path to proper install management).

A "scripting language", roughly speaking, is a language runtime where this code input is human-readable. A compiler places an extra step between the user and the runtime. "Joke" languages like Malbolge and APL needn't be human-readable in any form. But it's all the same thing at one level, and anyway human-readable does not mean that all potential users have the skills to read or write it, or can be expected to develop them.

There are good reasons why you don't normally expose a language runtime directly to end-users. The main one being that removing flexibility increases convenience.

If I want to type a SO post, I just want to type it. I'm perfectly capable of instead writing a C++ program to output it, but I would not use a web browser that exposed a C++ program editor instead of a regular text box. People who don't know C++ not only wouldn't use the browser, they couldn't.

If I want to configure certain business parameters then I don't necessarily want to do that using a Turing-complete specification language, and even if I did this is probably not distinguishable from "hard-coding" those same business parameters in any other programming language. You still need to consider whether what you're writing means what you want it to mean. You still need to test that changes are correct. That is, you still need programming skills for any tasks that are non-trivial and not anticipated by someone who does have programming skills who prepared a specialised sub-system ("application") for you to configure ("use").

So if you're about to embark on a 100% data-driven system, that can do anything given the right data, you have two questions to ask yourself:

1. Are we in the business of inventing programming languages, or should we be?
2. Will our new programming language be better (for our purposes) than the ones we already have and will we support and develop it as it needs?

Sometimes the answers are yes, and you write a domain-specific language of some kind. Or even a real general-purpose programming language if you're Sun/Microsoft/Stroustrup/van Rossum/many others. Sometimes the answers are no and you have the "inner platform" effect -- after much effort and trial and error you end up with something. If you're lucky it's only slightly inferior to the programming language you wrote it in, and no easier to use.

Some languages are harder or easier to use than others, in particular if they are specialised to a purpose like R then some users will find them much easier. What you probably won't do, is make general applications programming fundamentally easier. At any one time there are probably several people/organizations in the world with the potential to do that, but your boss/company has to honestly consider whether or not that includes him/you.

There's a trick often used for games, which is to expose Lua bindings to the game engine. This allows designers to program in a relatively easy language, but still engage a "real" programmer where necessary for performance or to access particular functionality of the engine or the platform. The resulting Lua scripts are "data" as far as the engine is concerned. They don't all need to include much of what you'd call "logic" as opposed to config data, and often they pretty much define all the plot and environment, but not the whole gameplay. This is not 100% data-driven and it certainly isn't 100% error-free, but it's an interesting practical compromise.

Answer 4

I worked at a company where this was the goal. Snippets of SQL were stored in database tables, read at runtime, and executed. Performance was terrible, as you can imagine, and bugs were frequent. It was also impossible to debug, with no stack traces or anything else that makes life easy.

"Data-driven programming" results from a fundamental lack of understanding of what we're doing, as programmers; any data that is capable of making an algorithm happen is actually "programming," even if you've somehow managed to mingle (mangle?) the two ideas in the user interface. Now, this doesn't mean that you can't combine the two ideas from the other direction, so that all code is data; that's the premise behind lisp, which is enabled by its homoiconicity and exploited by its macro system. Yes, these concepts sound similar, but their implications and applications are very different in practice.

Also, this may be editorializing, but the places I've encountered that want "completely data-driven" programming really don't value their programmers. They think of the code as a cost center, something to be outsourced, or ignored.

Answer 5

You mean your boss wants you to write this:

[
  {
    "statement": "Assignment ",
    "variable": "App",
    "value": {
      "type": "Function",
      "arguments": [],
      "function-body": [
        {}
      ]
    }
  },
  {
    "statement": "Assignment",
    "variable": "App.prototype.action",
    "value": {
      "type": "Function",
      "arguments": [
        "data"
      ],
      "function-body": [
        {
          "statement": "Call",
          "function-name": "console.log",
          "arguments": [
            "data"
          ]
        }
      ]
    }
  }
]

To generate this:

var App = function () {};
App.prototype.action = function ( data ) {
    console.log( data );
}

First one is JSON and the second one is JavaScript.

Clearification

I think theoretically it isn't because the thing that interprets the data ends up needing to become turing complete to describe the application so you've just shifted the problem one level higher to no net benefit.

Is such a 100% Data Driven Application possible?

This is where I just started. With my answer I'm trying to agree with the original post that: It is possible, but you're correct, it will just shift the problem one level higher for no [obvious] benefit.

Answer 6

You could argue convincingly, I think, that any web browser application could be considered 100% data driven¹.

Of course, that doesn't make it simpler or easier to build applications on the web, in fact it makes them much harder.

Tell your boss that he's reinventing the web browser, and he will eventually have to reinvent JavaScript to build anything reasonably complex.

¹ Well, if you ignore plugins, JavaScript and HTML5.

Answer 7

Yes. As far as I know, a system like Mathematica, which is a so-called powerful programming language but essentially is a shell, is built upon the similar idea that your boss has expected. Wolfram Mathematica is now becoming complex enough such that many computational tasks can be easily done by it.

Data driven is a concept. If we programmers are going to manipulate data in a simple way, we need a shell that is easy for us to play with data. Try to understand that once we start talking about learning a programming language based on syntax, we are actually learning the application interface or simply its shell. If we understand the shell, we can drive the programs.

As for 100 % data driven, if the compiler or the interpreter can understand the shell, computation is driven. If the data has the same underlying structure as the shell or interface that has, the data can be driven by compiler or interpreter too. I think Mathematica is a good explanation for why I respond you with a yes.