Steve Yegge's The Pinoccio Problem describes a very special type of program: one that not only fulfills the original purpose of its creators, but also is capable of performing arbitrary, user-defined computations.

They typically also host a console, by which one can reprogram the software on runtime, maybe persisting the modifications.

I find this problem very hard to reason about - there seems to be a conflict between implementing the 'core modules' of a program, and making the system really implementation-agnostic (i.e. no functionality is hard-coded).

So, how to architecture such a program - what techniques can help? Is it a well-studied topic?

  • 2
    It seems that I'm compelled to read another one of Steve Yegge's marathon blog posts. – Robert Harvey Nov 19 '12 at 19:28

A domain-specific language is usually the best answer to this kind of problem.

Some languages, such as Ruby or Boo, are well-designed to allow you to create DSLs. But this is a large subject, way more than can be covered in one answer here, but there are a couple of good books on the subject:

  • Maybe I'm missing the other half of the story, but it seems to me that DSLs are more of a convenience tool, e.g. one extends Emacs in elisp, which is a general-purpose language. – vemv Nov 19 '12 at 20:18
  • So you're saying that a DSL is something that you use to make an application extensible to the end-user, whose needs you do not know. Which is exactly what you were looking for, no? – pdr Nov 20 '12 at 1:37
  • @vemv: Yes, Emacs LISP is a complete programming language. The key is that Emacs models (buffers, windows, etc.) can be manipulated via Emacs LISP. We used to build a lot of applications this way using TCL. – kevin cline Nov 20 '12 at 18:22
  • @pdr DSLs and extensibity seem orthogonal to me - HTML is a DSL, but it has a rather limited scope of possibilities. – vemv Nov 21 '12 at 1:35
  • @kevincline great to hear. I just found out the architecture of Emacs is described in shop.oreilly.com/product/9780596517984.do - it should be a good read. – vemv Nov 21 '12 at 1:50

Most people don't know what they want, and those who do know what they want don't know how to describe it in a way that

  • a computer can understand,
  • is user-friendly enough for a mere mortal,
  • is robust enough that it doesn't crash every 5 minutes, and
  • is flexible enough to withstand future business changes.

That's why programmers exist.

Since users are not really interested in writing their own programs, what you wind up doing is writing a system that is configurable; that is, it has settings that are already understood by the system that can be manipulated by a user. If you are lucky, the number of such settings is small enough that the flexibility they provide is sufficient for the problem domain without adding too much complexity.

As pdr points out in his answer, Domain-Specific Languages are a good bridge for those folks who need more power and flexibility than configuration can provide.

  • OK, you think that Steve Yegge is wrong and we shouldn't expose the innards of our application in the way that Steve likes. That may or may not be true (for products aimed at programmers - like emacs, it often isn't), but does not help someone try to do what Yegge suggested. – btilly Nov 19 '12 at 22:52
  • I don't know if Steve is right or wrong; I haven't read the blog post yet. :) And I haven't really stated whether I'm for or against the practice you describe, but you're right: it works better if your customers are already programmers. For an example of this, check out Future Pinball, which includes a scripting language for determining the table behaviors. – Robert Harvey Nov 19 '12 at 22:54
  • +1: Good summary of the real issues involved in programming. Even specifying what's desired in a non-ambiguous way a human can understand is beyond the reach of most people! – R. Barzell Dec 4 '14 at 13:55

It is a studied topic, although unfortunately not a well studied one. Such systems are usually called Self-Sustaining Systems, Self-Supporting Systems or (especially in the Smalltalk community) Lively Systems.

Not surprisingly, Smalltalk itself is such a system, and the Smalltalk community (especially Dr. Alan Kay himself) are constantly striving to build ever more powerful and at the same time ever simpler such systems. Dr. Alan Kay's research group is currently working on designing a new language and new system that will provide a full Personal Computing System (i.e. Operating System, Programming Language, Compiler, VM, IDE, Editor, Office Suite, Web Browser, Realtime Collaboration etc.) in less than 20000 SLOC total.

  • 1
    Sounds cool, good luck to them! What is the project's name/page? – vemv Nov 19 '12 at 23:45

This sounds suspiciously like the "inner platform" (anti)pattern.

The basic idea is to write a piece of software that incorporates into itself a free-form means of configuration and/or command specification. This method is usually, as others have correctly pointed out, a "Domain-Specific Language" or DSL, interpreted by your software into more static pieces of code you have written. There are many such languages already out there; VBA, SQL, the scripting behind the Quake and Source engines, etc.

However, the flaw in such a pattern becomes evident when you look at the example DSLs. I doubt there's a DBA on the planet who would trust anyone in Accounting enough to give them a copy of SQL Management Studio and let them run their own reporting queries. On the flip side, I doubt there's any accountant on the planet who thinks that's a valid solution to their need for custom reports. 99% of users have no clue how to write a script to automate something in a Quake or Source-engine game; the ones that do keep finding holes in what those engines don't allow you to do, to do something that gives an unfair advantage.

While we could implement our own DSL (graphical or text-based) that simplifies the creation of custom queries and that does not allow the user to screw up anything beyond their own query, there is a very fine line to tread when designing a DSL that allows the user to do what they need, in a way they understand, without preventing then doing something they'd legitimately need, or allowing them to do something harmful. Often, there is no such solution, and even if it exists, providing a DSL that allows them to do anything they should be able to will not shield you from ever having to change the DSL to allow something new that nobody ever thought about before, that is perfectly acceptable and not harmful. As such, inner platform doesn't really solve the problem it was intended to; letting users do their own "programming", so the real developers can do more interesting things than customizing UI or report layouts/content. The devs will always be called on to "explain" (that is, hold the user's hand) the DSL they gave the user, and to answer the "well why can't I just do it like this" questions that are inevitable.

Even programmers deal with this all the time, in dealing with DSLs designed for interaction with third-party software. "Hey, why can't I do X, it sounds simple enough and I really need it to do this thing my user wants"... "Well if we let you do that then an attacker could do something similar and really screw you up, and us too". We as programmers expect a certain amount of that; end users expect a little more ease of use. That's why we're here.

  • Thanks for pointing out the 'inner platform' phenomenon. While the examples one can find are pretty ridiculous (e.g. wrapping SQL, HTTP), I can see how this can be a real peril. – vemv Nov 20 '12 at 0:13
  • -1: Your analogies are seriously flawed. The inner platform antipattern is an unnecessary layer that duplicates behavior already provided by the underlying system. A DSL is an intermediate layer that provides scriptable access to an underlying model. SQL is not a DSL for an accounting system built on a relational database. A set of stored procedures could provide such a DSL. – kevin cline Nov 20 '12 at 18:18
  • ... And those stored procedures would need constant maintenance work by the developers whenever the accountants want some new field or aggregate. A DSL developed to allow limited access to the more powerful engine underneath, which is what we're talking about, is by definition an inner platform; whether it's good or bad depends on whether the DSL is a solution looking for a problem (IOW there's a simpler way to give them the flexibility they need) and how well it's designed. Most are bad because they're unnecessary, or poorly designed (like limited-SQL in a reporting module). – KeithS Nov 20 '12 at 18:29

One of my favorite concepts in software engineering is convention over configuration and it came to mind reading your question. What I like to do is extract as much of app logic definition into some form of data representation, be it XML/json config files or database tables. I try to avoid hard-coding logic in the app tier and only have it interpret the instructions that are defined (initially by the developer, later modifiable by user) in the data tier. It is almost like a 4GL that your app controller understands. The non-technical user then has a friendly interface to manipulate that logic. This approach comes with a heightened degree of responsibility -- as you are giving the user a socket to change the application's workings inside out. So things like allowing test runs before the changes are committed, ACL, permissioning etc. are all critical in this approach.

In the future we will see more and more separation between logic definition and compilable code.

  • which could, i guess, be considered, to some degree, a DSL – amphibient Nov 19 '12 at 22:10

".... the fullness of time, I believe programming fluency will become as ubiquitous as literacy, so it won't matter....."

He believes everyone is destined to become a programmer, I disagree. In the real world, people have become less knowledgeable of the things they do daily as time has moved on. You used to turn a dial to tune a TV channel and understood "Frequency", and knew where each channel was, now you press the "Auto Program" button. You used to change your own oil and fix you own puncture when you owned a car, you had to know how to fix a breakdown, now you rarely have to do any of that, and take it to the garage or call AAA. You used to grow your own veges and cook your own food, now it's McD's and TV Dinners.....

I personally think that in the end programmers will program and users will use. The odd enthusiast will become an amateur programmer - but just like the real world, the tools they use will be substantially similar to what the professionals use. Imagine trying to build your own car with "Domain Specific Tools" that could only do up some of the nuts and bolts, or cook dinner with a "Safe and Easy to use" kitchen that had no knifes or hot surfaces, as they can do a lot of damage if used incorrectly.

  • Maybe my usage of 'user-defined' lead to some confusion. First, some apps are made for programmers and power-users rather than the mainstream. Also, scripts can be entirely optional (see Microsoft Office) or loaded 'indirectly' (JavaScript in a web browser). – vemv Nov 20 '12 at 0:35
  • +1: Amen! Programming complexity isn't about the language, it's about the task. Complexity, information flow, etc... won't go away with a new language or programming system, and making languages easy enough for the average person to use won't make the complexity of a task go away. Even if we systematize software development so it's a science, the formalisms required by that will be out of reach for most people. And most people probably don't want to do this anyway :) – R. Barzell Dec 4 '14 at 13:53

The answer to your question is bootstrapping. You first design a very simple, flexible and extensible language. (Dynamic languages work well.) You implement that. Then in that language you write extensions to the language until you've got the language that you want. Then you write the application itself in that language. When you're done, you expose the language that you started with.

Since everything is written in that language, and the extensions are available at runtime, you can do anything you want to the application (including unwise things).

For example see how emacs is written in eLisp. Or look at Smalltalk. Or Scheme.

That is the principle. The details tend to be very, very complicated. Therefore I highly recommend that you study code that seems to do this well. And read books like On Lisp that demonstrate how a language can be extended from within itself. If you've got the right kind of recursively twisted mind, eventually you'll be able to build your own system on this principle. (But good luck explaining to someone else what makes it cool...)

  • Yours is the answer that addresses my question most closely. I had to read the first paragraph several times though :) – vemv Nov 20 '12 at 0:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.