Do higher order functions violate the separation of data and code principle?

Question

I am asking here something that is really haunting for years (for real !).

I guess in a "perfect world", we would have programs made of data models and functions, and those functions manipulate data with a total separation betweend data and code.

But I've got the feeling that in practice it seems impossible to achieve because we're always facing cases in which we have to manipulate data models, structures, or higher-order objects rather than just "raw data".

It is a bit confusing for me. I think the problem is very simple to understand but maybe a little less to explain, so I'm going to take an example :

Consider you have a data model like this (pseudo-code) :

Class Thing
{
    int a;
    int b;
    string c;
}

Let's also consider we have a table containing rows of data following this data model, for instance :

+---+---+-------+
| a | b | c     |
+---+---+-------+
| 1 | 2 | "ghi" |
+---+---+-------+
| 3 | 5 | "abc" |
+---+---+-------+
| 7 | 1 | "xyz" |
+---+---+-------+
| 2 | 8 | "def" |
+---+---+-------+
| 4 | 3 | "tuv" |
+---+---+-------+

This could be internally represented by an array of objects, each object being an instance of our Thing class.

Now, we're in a very common situation where we want the end user able to see this data table on his screen and to sort rows according to a specific column. For instance if the table would be ordered according to the b field it would look like :

+---+---+-------+
| a | b | c     |
+---+---+-------+
| 7 | 1 | "xyz" |
+---+---+-------+
| 1 | 2 | "ghi" |
+---+---+-------+
| 4 | 3 | "tuv" |
+---+---+-------+
| 3 | 5 | "abc" |
+---+---+-------+
| 2 | 8 | "def" |
+---+---+-------+

This means the user can ask the table to be ordered according to something which is not data, since the b field is not data, it's an attribute/property. Which implies, somewhere in the program we have a function that is able to sort the table according to a specific field. Typically :

sortTable ( table , field , asc_or_desc )

But, here's the problem. If the table argument of the sorting function corresponds to actual data (since it represents all the data rows of the table), what is the field parameter ? Certainly not data. It refers to an attribute/property of the table's row data model. Functions are supposed to manipulate data, but here field is related to code/structure of the program, not data.

• What are the possibilities to overcome this ?
• Should we hardcode every possible ordering function (like sortTableByA, sortTableByB, ...) ?
• Should we give up on separation of data and code principle ?
• Like doing some mapping between strings (data) and class attributes ?
• Or using "evil" things like $$varname in php or obj.getattr(attrname) in python ?
• Or maybe just considering that data models are also data and thus we can use classes or attributes as function parameters just like any other data (but doesn't this raise a problem of program design by mixing layers) ?
• Or am I just missing something fundamental at some point ?

Thank you

Answer 1

I'll answer your question by completely denying all of your premises. :-)

I guess in a "perfect world", we would have programs made of data models and functions, and those functions manipulate data with a total separation between data and code.

That is a terrible world, not a perfect world. It's a terrible world because it is a world that unnecessarily prevents use of powerful tools.

What is computer programming, ultimately? Computer programming is the reification of thought. I have some idea for how to solve a problem, I write a program, and suddenly my idea is out there in the world, getting stuff done. (Or crashing horribly, but let's be optimistic.)

We are all good at reasoning at a high level of abstraction, and moreover, the higher the abstraction, the better we are at reasoning about it. There are people who program at the level of voltages, but I sure don't. There are people who program at the level of bits. And integers. There are people who program at the level of "if" statements and strings and objects. There are research languages where we can program at the level of interactions between random variables in a stochastic workflow.

How do we do this? We compose lower-level abstractions into solutions to problems at higher levels of abstraction. How do stochastic workflow languages do what they do? They treat code that is written at the if-and-goto level as data that can be manipulated programmatically.

I've got the feeling that in practice it seems impossible to achieve because we're always facing cases in which we have to manipulate data models, structures, or higher-order objects rather than just "raw data".

Your feeling is correct. We should not attempt to achieve your perfect world. We should attempt to achieve the opposite. We should make it as easy as possible to build more powerful abstractions. That's how we advance the state of the art.

We should not be building programs at the level of manipulating "raw data" any more than we should be designing cities at the level of the grains of sand and glue that make the concrete. Cities are human systems, not piles of concrete, and we analyze the performance of cities at the level of those systems. Programs are the same. We're building abstract systems, and we need to analyze their merits as complex systems.

Functions are supposed to manipulate data

Functions are supposed to compute a useful result.

I was considering that all data that should be manipulated in a utopic programming language have to be valid code in this programming language.

FYI such a language is called "homoiconic". Lisp is a good example. But you've said this slightly backwards. Normally we say "all programs have to be valid data in this language", not "all data has to be valid program". Programs are a very specific kind of data; data is not a specific kind of program.

Either data and code are clearly distinct, either "code = data" . There's no middle ground.

Of course there's a middle ground. Lisp and Scheme are fully homoiconic. C# is partly homoiconic; C# lets you programmatically manipulate expressions, but not statements or types. Python lets you change almost all the rules for name lookup on the fly; JavaScript allows you to do dynamic lookup via the with block; Scheme only allows static lookup. The history of programming languages is nothing but middle grounds. Why? Because all design is the process of finding a middle ground between many competing principles.

The fundamental block you've got here is dogma; you've expressed that you've switched from one dogma to another. When I am designing a language I explicitly throw out my dogmas. I'm not looking to design an OO language, or a functional language. I'm looking to design a useful language for my users, and if a dogma helps, I embrace it, and if it hurts, I throw it away. Your notion that there is supposed to be a separation between "code" and "data" and ne'er the twain shall meet, or the opposite, that give me full homoiconicity or naught, are dogmas that prevent you from writing and using powerful tools. Stop believing it! Write tools that get useful stuff done efficiently and safely.

Answer 2

The separation of data and code is not a principle but a paradigm. The idea is intimately associated with the software engineering era of the structured design methods (SADT, SA/SD, etc..) that were mainstream between 1960 and 1980. The ultimate beauty was to separate functions and data and document the software with nice data flow diagrams.

But a lot of other paradigms do no longer rely on such strong separation and have gained more traction:

• object oriented design, that considers objects, which bundle data with behavior;
• functional programming, where the frontier becomes even more blurry, becaus a function can process data or functions to produce other data or other functions.
• artificial intelligence, where the data of a trained neural net becomes part of the processing service.
• reflection allows code to analyse itself to adapt the behavior, and data description languages, high level queries or data models are more managed as data, despite they are more like code.

So in view of all this:

• Yes, it makes sense to give up strong separation of data and code, except for domains where it helps (e.g. highly secured systems where code should not be corruptable by data) or if the language imposes it.

• This gives full flexibility whether hardcoding every possible ordering function or constructing them dynamically; or using metadata in all form (mappings between strings and data fields, using reflection, etc.) instead creating additional code; in short, choose the approach that best fits the need.

Answer 3

Separation between data and behaviour sometimes makes sense, sometimes not. For example, OOP is the idea that objects contain all the behaviour necessary to manipulate their data. And functional programming is the idea that functions are ordinary data. Use whatever approach is appropriate towards the goals of your software.

What are the possibilities to overcome this?

You want your table to be sortable by different columns. To do this, you might want to introduce some kind of value that represents a column, for example its name as a string. Then: sortTable(t, "b", asc). That is a perfectly reasonable design, especially when the columns are created at runtime. If the columns are statically known, you might instead have an enum.

The alternative is a more functional approach where you provide some sorting key extractor function or comparator. This callback is given each row and returns something indicating how it should be sorted. For example: sortTable(t, row -> row.b, asc) or sortTable(t, (rowA, rowB) -> rowA.b <=> rowB.b, asc).

Should we hardcode every possible ordering function (like sortTableByA, sortTableByB, ...)?

That approach is usually not appropriate unless the columns are known in advance and only few columns exist – and if every callsite knows in advance which column it wants to sort. It might also be appropriate if the different columns behave very differently. It is usually simpler for everyone involved to represent the column as a value, e.g. as a string or enum. That would also lead to less repeated code in the sorting functions.

Should we give up on separation of data and code principle?

Perhaps. It depends.

Like doing some mapping between strings (data) and class attributes?

Yes, that can be a reasonable approach, especially in more dynamic settings.

Or using "evil" things like $$varname in php or obj.getattr(attrname) in python?

Those are not inherently evil. Those just are tools. Tools that are problematic when misused, but which can be indispensable in complicated scenarios.

Potential problems are that such approaches are difficult to debug, can lead to coupling and action at a distance, can hide errors because errors only manifest at runtime, and make encapsulation more difficult.

Or maybe just considering that data models are also data and thus we can use classes or attributes as function parameters just like any other data (but doesn't this raise a problem of program design by mixing layers)?

Sometimes it makes sense to keep things separate, sometimes not. Which approaches are appropriate also depends on the language you are using – there will be very different approaches for Python or C++. If your language doesn't offer certain things directly, it can make sense to define your own objects to represent these concepts.

Whether “mixing layers” is a problem depends on your overall design – it might also be the case that you are drawing your boundaries between layers in the wrong places, or that the entire concept of “layers” is not appropriate for your problem.

Or am I just missing something fundamental at some point ?

Not really. Difficult things are difficult. There isn't always one obvious best answer. There are also few absolute rules that must always be satisfied in software design – best practices like the Single Responsibility Principle are just heuristics that sometimes help arriving at a good design, and sometimes not.