"Everything is a Map", am I doing this right?

Question

I watched Stuart Sierra's talk "Thinking In Data" and took one of the ideas from it as a design principle in this game I'm making. The difference is he's working in Clojure and I'm working in JavaScript. I see some major differences between our languages in that:

• Clojure is idiomatically functional programming
• Most state is immutable

I took the idea from the slide "Everything is a Map" (From 11 minutes, 6 seconds to > 29 minutes in). Some things he says are:

1. Whenever you see a function that takes 2-3 arguments, you can make a case for turning it into a map and just passing a map in. There are a lot of advantages to that:
   1. You don't have to worry about argument order
   2. You don't have to worry about any additional information. If there are extra keys, that's not really our concern. They just flow through, they don't interfere.
   3. You don't have to define a schema
2. As opposed to passing in an Object there's no data hiding. But, he makes the case that data hiding can cause problems and is overrated:
   1. Performance
   2. Ease of implementation
   3. As soon as you communicate over the network or across processes, you have to have both sides agree on the data representation anyway. That's extra work you can skip if you just work on data.

3. Most relevant to my question. This is 29 minutes in: "Make your functions composable". Here's the code sample he uses to explain the concept:

   ;; Bad
   (defn complex-process []
     (let [a (get-component @global-state)
           b (subprocess-one a) 
           c (subprocess-two a b)
           d (subprocess-three a b c)]
       (reset! global-state d)))
   
   ;; Good
   (defn complex-process [state]
     (-> state
       subprocess-one
       subprocess-two
       subprocess-three))
   

   I understand the majority of programmers aren't familiar with Clojure, so I'll rewrite this in imperative style:

   ;; Good
   def complex-process(State state)
     state = subprocess-one(state)
     state = subprocess-two(state)
     state = subprocess-three(state)
     return state
   

   Here are the advantages:

   1. Easy to test
   2. Easy to look at those functions in isolation
   3. Easy to comment out one line of this and see what the outcome is by removing a single step
   4. Each subprocess could add more information on to the state. If subprocess one needs to communicate something to subprocess three, it's as simple as adding a key/value.
   5. No boilerplate to extract the data you need out of the state just so that you can save it back in. Just pass in the whole state and let the subprocess assign what it needs.

Now, back to my situation: I took this lesson and applied it to my game. That is, almost all of my high level functions take and return a gameState object. This object contains all the data of the game. EG: A list of badGuys, a list of menus, the loot on the ground, etc. Here's an example of my update function:

update(gameState)
  ...
  gameState = handleUnitCollision(gameState)
  ...
  gameState = handleLoot(gameState)
  ...

What I'm here to ask about is, have I created some abomination that perverted an idea that is only practical in a functional programming language? JavaScript isn't idiomatically functional (though it can be written that way) and it's really challenging to write immutable data structures. One thing that concerns me is he assumes that each of those subprocesses are pure. Why does that assumption need to be made? It's rare that any of my functions are pure (by that, I mean they often modify the gameState. I don't have any other complicated side effects other than that). Do these ideas fall apart if you don't have immutable data?

I'm worried that one day I'll wake up and realize this whole design is a sham and I've really just been implementing the Big Ball Of Mud anti-pattern.

---

Honestly, I've been working on this code for months and it's been great. I feel like I'm getting all the advantages he's claimed. My code is super easy for me to reason about. But I'm a one man team so I have the curse of knowledge.

Update

I've been coding 6+ months with this pattern. Usually by this time I forget what I've done and that's where "did I write this in a clean way?" comes into play. If I haven't, I'd really struggle. So far, I'm not struggling at all.

I understand how another set of eyes would be necessary to validate its maintainability. All I can say is I care about maintainability first and foremost. I'm always the loudest evangelist for clean code no matter where I work.

I want to reply directly to those that already have a bad personal experience with this way of coding. I didn't know it then, but I think we're really talking about two different ways of writing code. The way I've done it appears to be more structured than what others have experienced. When someone has a bad personal experience with "Everything is a map" they talk about how hard it is to maintain because:

1. You never know the structure of the map that the function requires
2. Any function can mutate the input in ways you'd never expect. You have to look all over the code base to find out how a particular key got into the map or why it disappeared.

For those with such an experience, perhaps the code base was, "Everything takes 1 of N types of maps." Mine is, "Everything takes 1 of 1 type of map". If you know the structure of that 1 type, you know the structure of everything. Of course, that structure usually grows over time. That's why...

There's one place to look for the reference implementation (ie: the schema). This reference implementation is code the game uses so it can't get out of date.

As for the second point, I don't add/remove keys to the map outside of the reference implementation, I just mutate what's already there. I also have a large suite of automated tests.

If this architecture eventually collapses under its own weight, I'll add a second update. Otherwise, assume everything is going well :)

Answer 1

I've supported an application where 'everything is a map' before. It's a terrible idea. PLEASE don't do it!

When you specify the arguments that are passed to the function, that makes it very easy to know what values the function needs. It avoids passing extraneous data to the function that just distracts th programmer - every value passed implies that it's needed, and that makes the programmer supporting your code have to figure out why the data is needed.

On the other hand, if you pass everything as a map, the programmer supporting your app will have to fully understand the called function in every way to know what values the map needs to contain. Even worse, it's very tempting to re-use the map passed to the current function in order to pass data to the next functions. This means that the programmer supporting your app needs to know all functions called by the current function in order to understand what the current function does. That's exactly the opposite of the purpose for writing functions - abstracting problems away so that you don't have to think about them! Now imagine 5 calls deep and 5 calls wide each. That's a hell of a lot to keep in your mind and a hell of a lot of mistakes to make.

"everything is a map" also seems to lead to using the map as a return value. I've seen it. And, again, it's a pain. The called functions need to never overwrite each other's return value - unless you know the functionality of everything and know that the input map value X needs to be replaced for the next function call. And the current function needs to modify the map to return it's value, which must sometimes overwrite the previous value and must sometimes not.

edit - example

Here's an example of where this was problematic. This was a web application. User input was accepted from the UI layer and placed in a map. Then functions were called to process the request. The first function set would check for erroneous input. If there was an error, the error message would be put in the map. The calling function would check the map for this entry and write the value in the ui if it existed.

The next function set would start the business logic. Each function would take the map, remove some data, modify some data, operate on the data in the map and put the result in the map, etc. Subsequent functions would expect results from prior functions in the map. In order to fix a bug in a subsequent function, you had to investigate all prior functions as well as a the caller to determine everywhere the expected value might have been set.

The next functions would pull data from the database. Or, rather, they'd pass a the map to the data access layer. The DAL would check if the map contained certain values to control how the query executed. If 'justcount' was a key, then the query would be 'count select foo from bar'. Any of the functions that was previously called might have ben the one that added 'justcount' to the map. The query results would be added to the same map.

The results would bubble up to the caller (business logic) which would check the map for what to do. Some of this would come from things that were added to the map by the initial business logic. Some would come from the data from the database. The only way to know where it came from was to find the code that added it. And the other location that can also add it.

The code was effectively a monolithic mess, that you had to understand in it's entirety to know where a single entry in the map came from.

Answer 2

Personally, I wouldn't recommend that pattern in either paradigm. It makes it easier to write initially at the expense of making it more difficult to reason about later.

For example, try to answer the following questions about each subprocess function:

• Which fields of state does it require?
• Which fields does it modify?
• Which fields are unchanged?
• Can you safely rearrange the order of the functions?

With this pattern, you can't answer those questions without reading the entire function.

In an object oriented language, the pattern makes even less sense, because tracking state is what objects do.

Answer 3

What you seem to be doing is, effectively, a manual State monad; what I would do is build a (simplified) bind combinator and re-express the connections between your logical steps using that:

function stateBind() {
    var computation = function (state) { return state; };
    for ( var i = 0 ; i < arguments.length ; i++ ) {
        var oldComp = computation;
        var newComp = arguments[i];
        computation = function (state) { return newComp(oldComp(state)); };
    }
    return computation;
}

...

stateBind(
  subprocessOne,
  subprocessTwo,
  subprocessThree,
);

You can even use stateBind to build the various subprocesses from subsubprocesses, and continue down a tree of binding combinators to structure your computation appropriately.

For an explanation of the full, unsimplified State monad, and an excellent introduction to monads in general in JavaScript, see this blog post.

Answer 4

So, there seems to be a lot of discussion between the effectiveness of this approach in Clojure. I think it might be useful to look at Rich Hickey's philosophy as to why he created Clojure to support data abstractions in this way:

Fogus: So once incidental complexities have been reduced, how can Clojure help solve the problem at hand? For example, the idealized object-oriented paradigm is meant to foster reuse, but Clojure is not classically object-oriented—how can we structure our code for reuse?

Hickey: I would argue about OO and reuse, but certainly, being able to reuse things makes the problem at hand simpler, as you are not reinventing wheels instead of building cars. And Clojure being on the JVM makes a lot of wheels—libraries—available. What makes a library reusable? It should do one or a few things well, be relatively self-sufficient, and make few demands on client code. None of that falls out of OO, and not all Java libraries meet this criteria, but many do.

When we drop down to the algorithm level, I think OO can seriously thwart reuse. In particular, the use of objects to represent simple informational data is almost criminal in its generation of per-piece-of-information micro-languages, i.e. the class methods, versus far more powerful, declarative, and generic methods like relational algebra. Inventing a class with its own interface to hold a piece of information is like inventing a new language to write every short story. This is anti-reuse, and, I think, results in an explosion of code in typical OO applications. Clojure eschews this and instead advocates a simple associative model for information. With it, one can write algorithms that can be reused across information types.

This associative model is but one of several abstractions supplied with Clojure, and these are the true underpinnings of its approach to reuse: functions on abstractions. Having an open, and large, set of functions operate upon an open, and small, set of extensible abstractions is the key to algorithmic reuse and library interoperability. The vast majority of Clojure functions are defined in terms of these abstractions, and library authors design their input and output formats in terms of them as well, realizing tremendous interoperability between independently developed libraries. This is in stark contrast to the DOMs and other such things you see in OO. Of course, you can do similar abstraction in OO with interfaces, for instance, the java.util collections, but you can just as easily not, as in java.io.

Fogus reiterates these points in his book Functional Javascript:

Throughout this book, I’ll take the approach of using minimal data types to represent abstractions, from sets to trees to tables. In JavaScript, however, although its object types are extremely powerful, the tools provided to work with them are not entirely functional. Instead, the larger usage pattern associated with JavaScript objects is to attach methods for the purposes of polymorphic dispatch. Thankfully, you can also view an unnamed (not built via a constructor function) JavaScript object as simply an associative data store.

If the only operations that we can perform on a Book object or an instance of an Employee type are setTitle or getSSN, then we’ve locked our data up into per-piece-of- information micro-languages (Hickey 2011). A more flexible approach to modeling data is an associative data technique. JavaScript objects, even minus the prototype machinery, are ideal vehicles for associative data modeling, where named values can be structured to form higher-level data models, accessed in uniform ways.

Although the tools for manipulating and accessing JavaScript objects as data maps are sparse within JavaScript itself, thankfully Underscore provides a bevy of useful opera‐ tions. Among the simplest functions to grasp are _.keys, _.values, and _.pluck. Both _.keys and _.values are named according to their functionality, which is to take an object and return an array of its keys or values...

Answer 5

The Devil's Advocate

I think this question deserves a devil's advocate (but of course I'm biased). I think @KarlBielefeldt is making very good points and I'd like to address them. First I want to say that his points are great.

Since he mentioned this isn't a good pattern even in functional programming, I'll consider JavaScript and/or Clojure in my replies. One extremely important similarity between these two languages is they're dynamically typed. I'd be more agreeable with his points if I were implementing this in a statically typed language like Java or Haskell. But, I'm going to consider the alternative to the "Everything is a Map" pattern to be a traditional OOP design in JavaScript and not in a statically typed language (I hope I'm not setting up a strawman argument by doing this, please let me know).

For example, try to answer the following questions about each subprocess function:

• Which fields of state does it require?

• Which fields does it modify?

• Which fields are unchanged?

In a dynamically typed language, how would you normally answer these questions? A function's first parameter may be named foo, but what is that? An array? An object? An object of arrays of objects? How do you find out? The only way I know of is to

1. read the documentation
2. look at the function body
3. look at the tests
4. guess and run the program to see if it works.

I don't think the "Everything is a Map" pattern makes any difference here. These are still the only ways I know of to answer these questions.

Also keep in mind that in JavaScript and most imperative programming languages, any function can require, modify and ignore any state it can access and the signature makes no difference: The function/method could do something with global state or with a singleton. Signatures often lie.

I'm not trying to set up a false dichotomy between "Everything is a Map" and poorly designed OO code. I'm just trying to point out that having signatures that take in less/more fine/coarse grained parameters doesn't guarantee you know how to isolate, setup and call a function.

But, if you would allow me to use that false dichotomy: Compared to writing JavaScript in the traditional OOP way, "Everything is a Map" seems better. In the traditional OOP way, the function may require, modify or ignore state that you pass in or state that you don't pass in. With this "Everything is a Map" pattern, you only require, modify or ignore state that you pass in.

• Can you safely rearrange the order of the functions?

In my code, yes. See my second comment to @Evicatos's answer. Perhaps this is only because I'm making a game though, I can't say. In a game that's updating 60x a second, it doesn't really matter if dead guys drop loot then good guys pick up loot or vice versa. Each function still does exactly what it's supposed to do regardless of the order they're run. The same data just gets fed into them at a different update calls if you swap the order. If you have good guys pick up loot then dead guys drop loot, the good guys will pick up the loot in the next update and it's no big deal. A human won't be able to notice the difference.

At least this has been my general experience. I feel really vulnerable admitting this publicly. Maybe considering this to be okay is a very, very bad thing to do. Let me know if I've made some terrible mistake here. But, if I have, it's extremely easy to rearrange the functions so the order is dead guys drop loot then good guys pick up loot again. It will take less time than the time it took to write this paragraph :P

Maybe you think "dead guys should drop loot first. It would be better if your code enforced that order". But, why should enemies have to drop loot before you can pick loot up? To me that doesn't make sense. Maybe the loot was dropped 100 updates ago. It's not necessary to check if an arbitrary bad guy has to pick up loot that's already on the ground. That's why I think the order of these operations is completely arbitrary.

It's natural to write decoupled steps with this pattern, but it's difficult to notice your coupled steps in traditional OOP. If I were writing traditional OOP, the natural, naive way of thinking is to make the dead guys drop loot return a Loot object that I have to pass into the good guys pick up loot. I wouldn't be able to reorder those operations since the first returns the input of second.

In an object oriented language, the pattern makes even less sense, because tracking state is what objects do.

Objects have state and it's idiomatic to mutate the state making its history just disappear... unless you manually write code to keep track of it. In what way is tracking state "what they do"?

Also, the benefits of immutability go way down the larger your immutable objects get.

Right, as I said, "It's rare that any of my functions are pure". They always only operate on their parameters, but they mutate their parameters. This is a compromise I felt I had to make when applying this pattern to JavaScript.

Answer 6

I have found that my code tends to end up structured like so:

• Functions that take maps tend to be larger and have side effects.
• Functions that take arguments tend to be smaller and are pure.

I didn't set out to create this distinction but that is often how it ends up in my code. I don't think using one style necessarily negates the other.

The pure functions are easy to unit test. The larger ones with maps get more into the "integration" test area since they tend to involve more moving parts.

In javascript, one thing that helps a lot is using something like Meteor's Match library to perform parameter validation. It makes it very clear what the function expects and can handle maps quite cleanly.

For example,

function foo (post) {
  check(post, {
    text: String,
    timestamp: Date,
    // Optional, but if present must be an array of strings
    tags: Match.Optional([String])
    });

  // do stuff
}

See http://docs.meteor.com/#match for more.

:: UPDATE ::

Stuart Sierra's video recording of Clojure/West's "Clojure in the Large" also touches on this subject. Like the OP, he controls side effects as part of the map so testing becomes much easier. He also has a blog post outlining his current Clojure workflow which seems relevant.

Answer 7

The main argument that I can think of against this practice is that it's very difficult to tell what data a function actually needs.

What that means is that future programmers in the codebase will have to know how the function being called works internally - and any nested function calls - in order to call it.

The more I think about it, the more your gameState object smells like a global. If that's how it's being used, why pass it around?

Answer 8

There is more fitting name for what you are doing than Big ball of mud. What you are doing is called the God object pattern. It does not look that way at first sight, but in Javascript there is very little difference between

update(gameState)
  ...
  gameState = handleUnitCollision(gameState)
  ...
  gameState = handleLoot(gameState)
  ...

and

{
  ...
  handleUnitCollision: function() {
    ...
  },
  ...
  handleLoot: function() {
    ...
  },
  ...
  update: function() {
    ...
    this.handleUnitCollision()
    ...
    this.handleLoot()
    ...
  },
  ...
};

Whether or not is it a good idea probably depends on the circumstances. But it is certainly in line with the Clojure way. One of the aims of Clojure is to remove what Rich Hickey calls "incidental complexity". Multiple communicating objects is certainly more complex than a single object. If you split functionality into multiple objects, you suddenly have to worry about communication and coordination and dividing responsibilities. Those are complications that are only incidental to your original goal of writing a program. You should see the Rich Hickey's talk Simple made easy. I think this is a very good idea.

Answer 9

I just sorta faced this topic earlier today while playing with a new project. I'm working in Clojure to make a Poker game. I represented face-values and suits as keywords, and decided to represent a card as a map like

{ :face :queen :suit :hearts }

I could just as well have made them lists or vectors of the two keyword elements. I don't know if it makes a memory/performance difference so I'm just going with maps for now.

In case I change my mind later though, I decided that most parts of my program should go through an "interface" to access the pieces of a card, so that the implementation detail is controlled and hidden. I've got functions

(defn face [card] (card :face))
(defn suit [card] (card :suit))

that the rest of the program uses. Cards get passed around to functions as maps, but the functions use an agreed-upon interface to access the maps and thus shouldn't be able to mess up.

In my program, a card will probably only ever be a 2-valued map. In the question, entire game state is passed around as a map. Game state will be a lot more complicated than a single card, but I don't think there is fault to raise about using a map. In an object-imperative language I could just as well have a single big GameState object and call its methods, and have the same problem:

class State
  def complex-process()
    state = clone(this) ; or just use 'this' below if mutation is fine
    state.subprocess-one()
    state.subprocess-two()
    state.subprocess-three()
    return state

Now it's object-oriented. Is there anything particularly wrong with it? I don't think so, you're just delegating work to functions which know how to handle a State object. And whether you're working with maps or objects, you should be wary of when to split it up into smaller pieces. So I say using maps is perfectly fine, as long as you use the same care you would use with objects.

Answer 10

From what (little) I've seen, using maps or other nested structures to make a single global immutable state object like this is fairly common in functional languages, at least the pure ones, especially when using the State Monad as @Ptharien'sFlame mentioend.

Two roadblocks to using this effectively that I've seen/read about (and other answers here have mentioned) are:

• Mutating a (deeply) nested value in the (immutable) state
• Hiding a majority of the state from functions which don't need it and just giving them the little bit they need to work on / mutate

There are a couple of different techniques / common patterns which can help alleviate these issues:

The first is Zippers: these let one traverse to and mutate state deep within an immutable nested hierarchy.

Another is Lenses: these let you focus into the structure to a specific location and read/mutate the value there. You can combine different lenses together to focus on different things, sort of like an adjustable property chain in OOP (where you can substitute variables for actual property names!)

Prismatic recently did a blog post on using this sort of technique, among other things, in JavaScript/ClojureScript, which you should check out. They use Cursors (which they compare to zippers) to window state for functions:

Om restores encapsulation and modularity using cursors. Cursors provide update-able windows into particular portions of the application state (much like zippers), enabling components to take references to only the relevant portions of the global state, and update them in a context-free manner.

IIRC, they also touch on immutability in JavaScript in that post.

Answer 11

Whether this is a good idea or not will really depend on what you're doing with the state inside those subprocesses. If I understand the Clojure example correctly, the state dictionaries that are being returned are not the same state dictionaries that are being passed in. They are copies, possibly with additions and modifications, that (I assume) Clojure is able to efficiently create because the functional nature of the language depends on it. The original state dictionaries to each function are not modified in any way.

If I'm understanding correctly, you are modifying the state objects you pass into your javascript functions rather than returning a copy, which means you are doing something very, very, different from what that Clojure code is doing. As Mike Partridge pointed out, this is basically just a global that you explicitly pass to and return from functions for no real reason. At this point I think it's simply making you think you're doing something that you actually aren't.

If you actually are explicitly making copies of the state, modifying it, and then returning that modified copy, then carry on. I'm not sure that's necessarily the best way to accomplish what you're trying to do in Javascript, but it is probably "closeish" to what that Clojure example is doing.

Answer 12

If you have a global state object, sometimes called a "god object", that is passed to each process, you end up confounding a number of factors, all of which increase coupling, while decreasing cohesion. These factors all impact long-term maintainability in a negative way.

Tramp Coupling This arises from passing data through various methods that have no need for almost all of the data, in order to get it to the place that can actually deal with it. This kind of coupling is similar to using global data, but can be more contained. Tramp coupling is the opposite of "need to know", which is used to localize effects and to contain the damage that one errant piece of code can have on the whole system.

Data Navigation Every sub-process in your example needs to know how to get to exactly the data that it needs, and it needs to be able to process it and perhaps construct a new global state object. That is the logical consequence of tramp coupling; the entire context of a datum is required in order to operate on the datum. Again, non-local knowledge is a bad thing.

If you were passing in a "zipper", "lens", or "cursor", as described in the post by @paul, that is one thing. You would be containing the access, and allowing the zipper, etc, to control reading and writing the data.

Single Responsibility Violation Claiming each of "subprocess-one", "subprocess-two" and "subprocess-three" have only a single responsibility, namely to produce a new global state object with the right values in it, is egregious reductionism. It's all bits in the end, isn't it?

My point here is that having all the major components of your game have all the same responsibilities as your game defeats the purpose of delegation and factoring.

Systems Impact

The major impact of your design is low maintainability. The fact that you can keep the entire game in your head says that you are very likely an excellent programmer. There are lots of things I have designed that I could keep in my head for the entire project. That is not the point of systems engineering, though. The point is to make a system that can work for something larger than one person can keep in his head at the same time.

Adding another programmer, or two, or eight, will cause your system to fall apart almost immediately.

1. The learning curve for god objects is flat (i.e., it takes a very long time to become competent in them). Each additional programmer will need to learn everything that you know, and keep it in their head. You will only ever be able to hire programmers that are better than you, assuming you can pay them enough to suffer through maintaining a massive god object.
2. Test provisioning, in your description, is white box only. You need to know every detail of the god object, as well as the module under test, in order to set up a test, run it, and determine that a) it did the right thing, and b) it didn't do any of 10,000 wrong things. The odds are greatly stacked against you.
3. Adding a new feature requires that you a) go through every subprocess and determine if the feature affects any code in it and vice versa, b) go through your global state and design the additions, and c) go through every unit test and modify it to check that no unit under test affected the new feature adversely.

Finally

1. Mutable god objects have been the curse of my programming existence, some of my own doing, and some that I have been trapped in.
2. The State monad doesn't scale. State grows exponentially, with all the implications for testing and operations that implies. The way we control state in modern systems is by delegation (partitioning of responsibilities) and scoping (restricting access to only a subset of the state). The "everything is a map" approach is the exact opposite of controlling state.