# Do higher order functions violate the separation of data and code principle?

I am asking here something that is really haunting for years (for real !).

I guess in a "perfect world", we would have programs made of data models and functions, and those functions manipulate data with a total separation betweend data and code.

But I've got the feeling that in practice it seems impossible to achieve because we're always facing cases in which we have to manipulate data models, structures, or higher-order objects rather than just "raw data".

It is a bit confusing for me. I think the problem is very simple to understand but maybe a little less to explain, so I'm going to take an example :

Consider you have a data model like this (pseudo-code) :

``````Class Thing
{
int a;
int b;
string c;
}
``````

Let's also consider we have a table containing rows of data following this data model, for instance :

``````+---+---+-------+
| a | b | c     |
+---+---+-------+
| 1 | 2 | "ghi" |
+---+---+-------+
| 3 | 5 | "abc" |
+---+---+-------+
| 7 | 1 | "xyz" |
+---+---+-------+
| 2 | 8 | "def" |
+---+---+-------+
| 4 | 3 | "tuv" |
+---+---+-------+
``````

This could be internally represented by an array of objects, each object being an instance of our `Thing` class.

Now, we're in a very common situation where we want the end user able to see this data table on his screen and to sort rows according to a specific column. For instance if the table would be ordered according to the `b` field it would look like :

``````+---+---+-------+
| a | b | c     |
+---+---+-------+
| 7 | 1 | "xyz" |
+---+---+-------+
| 1 | 2 | "ghi" |
+---+---+-------+
| 4 | 3 | "tuv" |
+---+---+-------+
| 3 | 5 | "abc" |
+---+---+-------+
| 2 | 8 | "def" |
+---+---+-------+
``````

This means the user can ask the table to be ordered according to something which is not data, since the `b` field is not data, it's an attribute/property. Which implies, somewhere in the program we have a function that is able to sort the table according to a specific field. Typically :

``````sortTable ( table , field , asc_or_desc )
``````

But, here's the problem. If the `table` argument of the sorting function corresponds to actual data (since it represents all the data rows of the table), what is the `field` parameter ? Certainly not data. It refers to an attribute/property of the table's row data model. Functions are supposed to manipulate data, but here `field` is related to code/structure of the program, not data.

• What are the possibilities to overcome this ?
• Should we hardcode every possible ordering function (like sortTableByA, sortTableByB, ...) ?
• Should we give up on separation of data and code principle ?
• Like doing some mapping between strings (data) and class attributes ?
• Or using "evil" things like `\$\$varname` in php or `obj.getattr(attrname)` in python ?
• Or maybe just considering that data models are also data and thus we can use classes or attributes as function parameters just like any other data (but doesn't this raise a problem of program design by mixing layers) ?
• Or am I just missing something fundamental at some point ?

Thank you

• You write this as if the "separation of data and code" is desirable - why? The interchangeability of code and data was considered as one of the major strengths of LISP over the last 60 years, and yes, higher order functions are a indeed the perfect example for this. There is a reason why more and more functional tools found their way into all modern programming languages. Commented Sep 25, 2019 at 19:25
• What makes you think there is even a difference between data and code? Have you ever thought about what an interpreter or compiler does? I mean, really, deeply, philosophized about it? Commented Sep 25, 2019 at 19:29
• In fact, I did really, deeply philosophized about it, and this seems to be a dead end. This is something I'm thiking about for years At some point I add a totally different persepective : I was considering that all data that should be manipulated in a utopic programming language have to be valid code in this programming language. Which means the language has the highest level of introspection. If we consider a theoretical / philosophical point of view, things have to be clear : either data and code are clearly distinct, either "code = data" (my past way of thinking). There's no middle ground Commented Sep 25, 2019 at 20:18

I guess in a "perfect world", we would have programs made of data models and functions, and those functions manipulate data with a total separation between data and code.

That is a terrible world, not a perfect world. It's a terrible world because it is a world that unnecessarily prevents use of powerful tools.

What is computer programming, ultimately? Computer programming is the reification of thought. I have some idea for how to solve a problem, I write a program, and suddenly my idea is out there in the world, getting stuff done. (Or crashing horribly, but let's be optimistic.)

We are all good at reasoning at a high level of abstraction, and moreover, the higher the abstraction, the better we are at reasoning about it. There are people who program at the level of voltages, but I sure don't. There are people who program at the level of bits. And integers. There are people who program at the level of "if" statements and strings and objects. There are research languages where we can program at the level of interactions between random variables in a stochastic workflow.

How do we do this? We compose lower-level abstractions into solutions to problems at higher levels of abstraction. How do stochastic workflow languages do what they do? They treat code that is written at the if-and-goto level as data that can be manipulated programmatically.

I've got the feeling that in practice it seems impossible to achieve because we're always facing cases in which we have to manipulate data models, structures, or higher-order objects rather than just "raw data".

Your feeling is correct. We should not attempt to achieve your perfect world. We should attempt to achieve the opposite. We should make it as easy as possible to build more powerful abstractions. That's how we advance the state of the art.

We should not be building programs at the level of manipulating "raw data" any more than we should be designing cities at the level of the grains of sand and glue that make the concrete. Cities are human systems, not piles of concrete, and we analyze the performance of cities at the level of those systems. Programs are the same. We're building abstract systems, and we need to analyze their merits as complex systems.

Functions are supposed to manipulate data

Functions are supposed to compute a useful result.

I was considering that all data that should be manipulated in a utopic programming language have to be valid code in this programming language.

FYI such a language is called "homoiconic". Lisp is a good example. But you've said this slightly backwards. Normally we say "all programs have to be valid data in this language", not "all data has to be valid program". Programs are a very specific kind of data; data is not a specific kind of program.

Either data and code are clearly distinct, either "code = data" . There's no middle ground.

Of course there's a middle ground. Lisp and Scheme are fully homoiconic. C# is partly homoiconic; C# lets you programmatically manipulate expressions, but not statements or types. Python lets you change almost all the rules for name lookup on the fly; JavaScript allows you to do dynamic lookup via the `with` block; Scheme only allows static lookup. The history of programming languages is nothing but middle grounds. Why? Because all design is the process of finding a middle ground between many competing principles.

The fundamental block you've got here is dogma; you've expressed that you've switched from one dogma to another. When I am designing a language I explicitly throw out my dogmas. I'm not looking to design an OO language, or a functional language. I'm looking to design a useful language for my users, and if a dogma helps, I embrace it, and if it hurts, I throw it away. Your notion that there is supposed to be a separation between "code" and "data" and ne'er the twain shall meet, or the opposite, that give me full homoiconicity or naught, are dogmas that prevent you from writing and using powerful tools. Stop believing it! Write tools that get useful stuff done efficiently and safely.

• This answer is absolutely wonderful ! I agree when you say "Computer programming is the reification of thought". My current goal is to design a programming language with a paradigm which is the closest to the human brain's way of thinking : in term of concepts. My first attempts were homoiconic language but my actual vision is more like : code (= concept definitions) is like hardware, as if writing a program was like building a machine. What bothered me with homoiconicity was the fact that if would require a 'concept' concept and also to allow defining new concepts at runtime. Commented Sep 26, 2019 at 6:32
• @ibi0tux: Glad you like it. I certainly don't want to discourage you from pursuing your language investigations, but I would suggest that it might go easier if you have a more concrete use case in mind than "capture the language of thought". :-) That sounds hard! As for your struggles with "how do we extend the concept concept" and so on, I would suggest first that you look at the history of propositional logic, predicate logic, second-order logic, higher-order logic, and see how mathematicians have struggled with similar conceptual problems. Commented Sep 26, 2019 at 6:50
• Actually I don't have the pretension to "capture the language of thought" ;) I just meant my approach is to start from our way of thinking to create a language as close as possible to it, instead of starting from the machine and adding layers and layers of abstractions. The other aspects I am looking for are : (obviously) Turing-completeness, strong and strict typing (because concepts have to be auto-sufficient) and allow as much as possible formal verification. In this context, I don't think relfection and homoiconicity will make things easier. Commented Sep 26, 2019 at 7:51
• @ibi0tux: Though that philosophical question is interesting, what I would focus on is why we care about "types" at all. The point of adding types to programming languages is to make programs easier to understand and easier to be seen to be correct; a language which actively prevents incorrect programs is said to be "type safe". Type systems are therefore created in order to express different kinds of type safety. In C# for instance we've decided that "don't allow a giraffe to be passed to a method that expects a turtle" to be the most important factor in type safety. Commented Oct 1, 2019 at 21:25
• @ibi0tux: Put another way: deciding what kind of programs should be easy to write in your language is important, but do not neglect the equally important factor of deciding what kinds of programs should be hard to write in your language. C#'s designers decided early on that programs that crash the runtime should be hard to write, but not impossible. Commented Oct 1, 2019 at 22:04

The separation of data and code is not a principle but a paradigm. The idea is intimately associated with the software engineering era of the structured design methods (SADT, SA/SD, etc..) that were mainstream between 1960 and 1980. The ultimate beauty was to separate functions and data and document the software with nice data flow diagrams.

But a lot of other paradigms do no longer rely on such strong separation and have gained more traction:

• object oriented design, that considers objects, which bundle data with behavior;
• functional programming, where the frontier becomes even more blurry, becaus a function can process data or functions to produce other data or other functions.
• artificial intelligence, where the data of a trained neural net becomes part of the processing service.
• reflection allows code to analyse itself to adapt the behavior, and data description languages, high level queries or data models are more managed as data, despite they are more like code.

So in view of all this:

• Yes, it makes sense to give up strong separation of data and code, except for domains where it helps (e.g. highly secured systems where code should not be corruptable by data) or if the language imposes it.

• This gives full flexibility whether hardcoding every possible ordering function or constructing them dynamically; or using metadata in all form (mappings between strings and data fields, using reflection, etc.) instead creating additional code; in short, choose the approach that best fits the need.

Separation between data and behaviour sometimes makes sense, sometimes not. For example, OOP is the idea that objects contain all the behaviour necessary to manipulate their data. And functional programming is the idea that functions are ordinary data. Use whatever approach is appropriate towards the goals of your software.

What are the possibilities to overcome this?

You want your table to be sortable by different columns. To do this, you might want to introduce some kind of value that represents a column, for example its name as a string. Then: `sortTable(t, "b", asc)`. That is a perfectly reasonable design, especially when the columns are created at runtime. If the columns are statically known, you might instead have an enum.

The alternative is a more functional approach where you provide some sorting key extractor function or comparator. This callback is given each row and returns something indicating how it should be sorted. For example: `sortTable(t, row -> row.b, asc)` or `sortTable(t, (rowA, rowB) -> rowA.b <=> rowB.b, asc)`.

Should we hardcode every possible ordering function (like sortTableByA, sortTableByB, ...)?

That approach is usually not appropriate unless the columns are known in advance and only few columns exist – and if every callsite knows in advance which column it wants to sort. It might also be appropriate if the different columns behave very differently. It is usually simpler for everyone involved to represent the column as a value, e.g. as a string or enum. That would also lead to less repeated code in the sorting functions.

Should we give up on separation of data and code principle?

Perhaps. It depends.

Like doing some mapping between strings (data) and class attributes?

Yes, that can be a reasonable approach, especially in more dynamic settings.

Or using "evil" things like \$\$varname in php or obj.getattr(attrname) in python?

Those are not inherently evil. Those just are tools. Tools that are problematic when misused, but which can be indispensable in complicated scenarios.

Potential problems are that such approaches are difficult to debug, can lead to coupling and action at a distance, can hide errors because errors only manifest at runtime, and make encapsulation more difficult.

Or maybe just considering that data models are also data and thus we can use classes or attributes as function parameters just like any other data (but doesn't this raise a problem of program design by mixing layers)?

Sometimes it makes sense to keep things separate, sometimes not. Which approaches are appropriate also depends on the language you are using – there will be very different approaches for Python or C++. If your language doesn't offer certain things directly, it can make sense to define your own objects to represent these concepts.

Whether “mixing layers” is a problem depends on your overall design – it might also be the case that you are drawing your boundaries between layers in the wrong places, or that the entire concept of “layers” is not appropriate for your problem.

Or am I just missing something fundamental at some point ?

Not really. Difficult things are difficult. There isn't always one obvious best answer. There are also few absolute rules that must always be satisfied in software design – best practices like the Single Responsibility Principle are just heuristics that sometimes help arriving at a good design, and sometimes not.

• "That is a perfectly reasonable design, especially when the columns are created at runtime. If the columns are statically known, you might instead have an enum." Here's the whole point. If we columns are created at runtime, they should be data, since only data "should" be manipulated at runtime (ideally). But in this case columns are class attributes and this is precisely the problem. In a data-oriented paradigm, how to keep data data and code code ? Is it inevitable to resort to any form of mapping between data and code at some point ? Commented Sep 25, 2019 at 18:55
• @ibi0tux Callback-based sorting functions generally work perfectly fine when sorting by some object attribute – you don't have to represent the attribute if you don't want to. I don't think higher order functions are problematic. But representing the attributes as some value also seems perfectly fine. In Python just using strings might be the best approach. In Java I might use `enum Attribute { A, B, C }` and write a switch within the sorting function. In C++ I would consider member pointers.
– amon
Commented Sep 25, 2019 at 19:20
• Python usage of strings is exactly what I would like to avoid since strings are data. But, if we don't use data, our sorting function will become an higher-level function (which is not especially a problem), but still we need either to make a mapping, or consider that the end user will manipulate objects that are not data-level. Commented Sep 25, 2019 at 19:45
• @ibi0tux It's good to avoid stringly-typed APIs, but it's also good to be pragmatic. Those callback-based approaches are effectively standard, so just use that if your API exposes objects that represent rows in the table. If you don't want to expose row objects, choose some representation for the attributes, e.g. an enum. Python has enums as well!
– amon
Commented Sep 25, 2019 at 20:26
• Prag...what ? My question was originally abstracted from any existing programming language and purely theoretical (or even philosophical). Exposing objects that represent rows means the API that would do that turns data models into data once again. It is just a different way to achieve what I would like to avoid. It seems there's just no way to really separate code and data, even in a such ordinary and simple example ... :) Commented Sep 25, 2019 at 21:04