74

In this 2003 article by Stephen Figgins on linuxdevcenter.com, Bram Cohen's BitTorrent is described as using the "Fix Everything" design pattern.

A less common approach that both makes BitTorrent harder to grasp, but worthy of study, is Cohen's use of idempotence. A process is idempotent when applying it more than once causes no further changes. Cohen says he uses a design pattern he calls "Fix Everything," a function that can react to a number of changes without really noting what all it might change. He explains, "you note the event that happened, then call the fix everything function which is written in this very idempotent manner, and just cleans up whatever might be going on and recalculates it all from scratch." While idempotence makes some difficult calculations easier, it makes things a little convoluted. It's not always clear what a call is going to change, if anything. You don't need to know in advance. You are free to call the function, just to be on the safe side.

This sounds quite nice on the face of it.

However, it seems to me that calling an idempotent "fix everything" function would improve robustness of the system at the cost of efficiency and potentially screwing up the containing system (that might prefer processes that carefully plan and execute.).

I can't say that I've used it before, though. I also cannot find the source for his application online (but I did find this one that claims to be based on it.). Nor can I find reference to it outside of this article (and I consider my google-fu to be pretty good) but I did find an entry for "Idempotent Capability" on SOApatterns.org.

Is this idea better known by another name?

What is the "Fix Everything" design pattern? What are its pros and cons?

  • 4
    I suspect the name is also a reference to the idea of the fixed point of a function, x = f(x). No matter how many times you apply f to x, the result is the same. Once you achieved the correct result, re-processing the correct result returns the same correct result. – 9000 Apr 20 '17 at 15:05
  • 7
    Note that anyone can give any name to anything they want, but that doesn't make it a well-known software pattern. Idempotency is a well-known concept in its own right; it just looks like it's being used creatively here. – Robert Harvey Apr 20 '17 at 15:59
  • 1
    This reminds me of how the Main Event Loop was implemented on Mac OS. It was a single function that responded to any Event and was generally structured to test the state of all of the controls and update the entire UI as needed. Idempotent, indeed. – Lucas Apr 20 '17 at 17:26
  • 3
    This sounds quite nice on the face of it. Really? It sounds horrible to me! – Michael Apr 20 '17 at 18:05
  • 3
    @Michael You don't like package managers? They work by the same concept, just on a smaller scale: Mark what you want to system to look like, run "fix everything", it installs/removes/upgrades as appropriate, but only has anything to do if there were changes. – Izkata Apr 20 '17 at 18:44
101

Let's say you have an HTML page that is fairly complicated-- if you pick something in one dropdown, another control might appear, or the values in a third control might change. There's two ways you could approach this:

  1. Write a separate handler, for each and every control, that respond to events on that control, and updates other controls as needed.

  2. Write a single handler that looks at the state of all the controls on the page and just fixes everything.

The second call is "idempotent" because you can call it over and over again and the controls will always be arranged properly. Whereas the first call(s) may have issues if a call is lost or repeated, e.g. if one of the handlers performs a toggle.

The logic for the second call would be a bit more obscure, but you only have to write one handler.

And you can always use both solutions, calling the "fix everything" function as needed "just to be on the safe side."

The second approach is especially nice when state can come from different sources, e.g. from user input versus rendered from the server. In ASP.NET, the technique plays very well with the concept of postback because you just run the fix everything function whenever you render the page.

Now that I've mentioned events being lost or repeated, and getting state from different sources, I'm thinking it is obvious how this approach maps well to a problem space like BitTorrent's.

Cons? Well the obvious con is that there is a performance hit because it is less efficient to go over everything all the time. But a solution like BitTorrent is optimized to scale out, not scale up, so it's good for that sort of thing. Depending on the problem you are trying to solve, it might not be suitable for you.

  • 9
    It seems to me that MVC is typical of "Fix Everything": when you modify the model then redraw the view from scratch, the view is just fully redrawn, without attempting to divine what parts the action could potentially have affected. – Matthieu M. Apr 20 '17 at 11:18
  • 3
    This sounds essentially like the principle behind systems like Saltstack, Ansible, and Nix. Given a description of a configuration, you can theoretically bring several diverse systems into the same end state. – kojiro Apr 20 '17 at 11:31
  • 1
    @MatthieuM. React, which is quite popular on frontend development, is like that except it does virtual dom diffing so it only updates the real dom with the actual changes – Izkata Apr 20 '17 at 19:08
  • 2
    @Izkata Even more than React, this answer made me think of Redux. – Kevin Apr 20 '17 at 20:01
  • 2
    It might be useful to point out that "fix everything" and idempotent are different things: "fix everything" is usually idempotent (but doesn't need to be), and idempotent operations don't need to fix everything or even have a performance penalty - just give the same result when executed twice. – Hans-Peter Störr Apr 25 '17 at 6:25
15

I think the article is a bit dated because as I read it, this isn't really a unorthodox or new idea at all. This idea is presented as a separate pattern when it really is just a simple Observer implementation. Thinking back to what I was doing at the time, I remember working on logic to sit behind a somewhat complex interface with a number of different panels with data that were interdependent. The user could change values and/or run a optimization routine and based on those actions, events were generated that the UI would listen to and update as needed. There were a number of issues during development where certain panels wouldn't update when they should. The fix (staying within the design) was to generate events from other events. Ultimately, by the time everything was working right, almost every change resulted in all the panels to refresh. All the complexity of trying to isolate when a given panel needed to refresh was for naught. And it didn't matter anyway. It was effectively a premature optimization. I would have saved a ton of time and effort by simply collapsing it all into a single event that refreshed everything.

There are innumerable systems designed in the "fix everything" or refresh everything way. Think of all the CRUD interfaces that add/update a row and then requery the DB. This isn't an exotic approach, it's just the obvious non-clever solution. You have to realize that in 2003, it was the height of 'pattern fever'. From what I could tell, people thought naming new patterns was going to be their path to fame and riches. Don't get me wrong, I think the concept of a pattern is an extremely useful one for describing solutions in the abstract. Things just kind of went off the rails a bit. It's unfortunate because it created a lot of cynicism about the pattern concept in general. It's only in this context that it makes sense to talk about this as an 'unorthodox' solution. It's similar to the orthodoxy around ORMs or DI containers. Not using them is seen as unorthodox even though people had been building software long before these tools existed and in many cases those tools are overkill.

So back to 'fix everything'. A simple example is calculating means. The simple solution is to sum numbers and divide by the cardinality of the values. If you add or modify a number, you just do it again, from the beginning. You could keep track of the sum and the count of numbers and when someone adds a number, you increase the count and add it to the sum. Now you aren't re-adding all the numbers again. If you've ever worked with Excel with a formula that references a range and modified a single value in that range, you have an example of the 'fix everything' pattern i.e. any formula that has a reference to that range will recalculate regardless of whether that value was relevant (e.g. using something like sumif()).

This isn't to say this isn't a smart choice in a given context. In the mean example, lets say we now need to support updates. Now I need to know the old value somehow and only change the sum by the delta. None of this is really that challenging until you consider trying to do this in a distributed or concurrent environment. You now have to handle all kinds of thorny timing issues and you'll likely end up creating a major bottleneck which slows down things far more than recalculating.

The upshot here is that 'fix everything' or 'refresh everything' approach is much easier to get right. You can make a more sophisticated approach work but it's a lot more complicated and therefore more likely to be flawed. In addition in a lot of contexts, the 'refresh everything' approach can be more efficient. For example, copy on write approaches are generally slower for single-threaded approaches but when you have high concurrency, it can allow you to avoid locks and therefore provide better performance. In other cases, it can allow you to batch changes together in an efficient way. So for most problems, you probably want to start with a refresh everything approach unless you have a specific reason why you can't do it and then worry about doing something more complex once you have a need. A working implementation that you can regression test against is valuable in itself, even if it's slow.

  • 2
    I'm pretty sure Excel intends to only recalculate cells that are dependent on changes, which is why there's a way to trigger all cells to recalculate: superuser.com/questions/448376/… (which I suppose would be a "fix everything") – Aaron Hall Apr 19 '17 at 17:24
  • @AaronHall If it does, it's a really bad implementation. I regularly watch it consume 100% of 7 CPUs for 15-30 minutes to calculate e.g. 60,000 cells. The calculations aren't complicated. I've often written Python programs that can do everything in the sheet in a few seconds including starting Python. This was my best guess at how it could take so long. It could be something else I suppose. There are also a number of really old bugs in Excel which might be the reason for that feature. – JimmyJames Apr 19 '17 at 17:31
  • 1
    @AaronHall it's also possible with that user that auto-calculate was disabled on the sheet. I often do this on large workbooks because I don't have 15 minutes to spare every time I hit enter. – JimmyJames Apr 19 '17 at 17:34
  • @AaronHall I thought a little more and you have a point. My assumptions were likely overly broad. I updated the answer to me more focused on something I am more confident in. – JimmyJames Apr 19 '17 at 20:47
  • 2
    @JimmyJames: The point I meant to make is that the best approach can vary a lot depending upon circumstances, and "fix everything" can be subdivided into "eagerly fix everything on each individual change" and "lazily fix everything after all changes are complete". – supercat Apr 20 '17 at 20:38
4

Not sure it's a "design pattern", but I would classify that type of behavior as end state configuration or desired state configuration, in the vein of Puppet, Chef or Powershell DSC.

Those solutions typically operate at the systems management level, not a business logic level as the question describes, but it's the effectively the same paradigm, and although such tools are usually declarative in nature, the same principles can be applied in procedural code or scripting.

1

I have mostly used this within user interfaces. The nice thing is that you write it once, and it handles everything from the simplest to the hardest case equally well (for example if the user rotates the screen, or on a laptop/desktop if the user resizes a window, and virtually everything changes).

There isn't much reason to worry about efficiency. In the user interface, the expensive things are things like redrawing an item that has been moved. The calculation where each item goes and how big it is is usually quite fast. All you have to make sure of is that whenever you find that an item should stay in exactly the place where it belongs, no code is executed to move it. The real changes are all things that you had to do anyway.

0

Sounds like reactive programming principles. "Fix everything" looks at a the current "core" state and propagates everything else that should be affected - "computed states". If you optimize this derivation it may reach high efficiency, a-la React, if done naively performance may not be optimal although it could still be fast enough.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.