How can we avoid extremely complex configurations in enterprise software?

Question

This is the situation that I've seen two times in a row already.

A company makes software intended to be sold to other companies. So there will be relatively few clients, but each is an important one. An obvious, though by far not exhaustive example of such software is an ERP system.

Now each client company has their own peculiar workflow and their own peculiar requirements. It is the product that must adapt to the client, not the other way around, or else the client will not buy the product.

The result is that there will be multiple, similar though different versions of the software, each version tailored to the needs of each particular client.

From my experience, the common solution to this conundrum is to attempt to make the software highly general and configurable. An entire team will be dedicated to understand the very complex configuration system and interact with the clients. This team will be responsible to click their way through this very complex configuration system, thus configuring the software for the client, adapting it to the client's needs and readying it to work.

This approach, from my experience, has important flaws.

• From the programmers' point of view, the product, in its entirety, is incoherent, too big, too complex and frankly weird. It is hard to understand it and develop it.
• As mentioned before, the configuration is really very complex. Being able to understand it and to use it properly is not a small feat, which is why there must be employees who specialize in this very task. The configuration is also dangerous, since any mistake in clicking the numerous checkboxes, selecting proper values from numerous lists, sometimes even entering pieces of code, etc., etc. will result in the whole software working incorrectly, maybe even causing data corruption.

Can this be somehow avoided?

I wonder if the famed article on the Daily WTF about Soft Coding is not the answer here. Indeed, the configuration becomes so vast that I wonder if (even if the possibility to enter custom code is ignored) it is not accidentally Turing-complete. This is because in this approach, effectively, the configuration will have to store business rules. But business rules belong in code, not configuration, and the code is where they should be modified and adapted.

It is obvious that the software must adapt to the client, not the other way around. But perhaps the proper solution is to ditch the utopic goal to make the software so very general and configurable, strip the configuration system to the bare minimum and embrace that the product will be adapted to each client through code, rather than configuration?

Maybe simply utilize conditional compilation? The product will be compiled and deployed separately for each client. Define compile-time constants denoting the client, enable or disable features in the code rather than configuration depending on the client.

The counterargument to the above is that if that was embraced, then the software engineers will become responsible to directly respond to clients' needs. Each time a client has any grievance a patch will have to be made in the code, the whole system will have to be compiled, tested, and deployed again. This will take days, if not weeks.

Such delays are unacceptable unless absolutely necessary. Whenever possible the clients' needs must be answered by sending our configuration specialist, who will then reconfigure the product within a few clicks. This will take just one day.

But, wasn't this fallacy addressed in the aforementioned The Daily WTF article? This is what The Daily WTF calls "The Dreaded Deployment". But if making changes in code and deploying is hard, then the solution is to fix the issues that make deployment unwieldy, rather than trying to avoid deployments by letting business rules be set via configuration.

Maybe, after all, there aren't any hard reasons why making changes in the code and redeploying the product must take days, if not weeks?

Also, from my experience, frequent changes in the code to respond to changing business requirements is a reality any way and the very complex configuration doesn't remove this need.

Why do companies making software sold to other companies tend to attempt to make the code general enough to meet the needs of all clients at the same time, while adapting the product to the needs of each client through configuration?

Is there a better way than that to let the product adapt to the clients?

Would stripping configuration to the bare minimum and adapting the product to each client in the code help here? Would it even be feasible?

Answer 1

Why do companies making software sold to other companies tend to attempt to make the code general enough to meet the needs of all clients at the same time, while adapting the product to the needs of each client through configuration?

Profit.

This is an attempt to minimize cost. Everyone wants a custom fit at off-the-rack prices. But if you don't hire a tailor, the rack is what you get.

The problem is we haven't been very good at standardizing. And so companies get away with insisting they need a shirt with 3 arm holes.

So you get salespeople pretending to be tailors. Oh sure they don't use a needle. Just scissors and a button gun. So they aren't really tailors right?

Sorry but it really is just people solving the same problems using different tools.

The heart of the problem is the customizations don't fit into any design that anticipated them. So your lucky if the holes that get cut don't unravel the whole garment. And that goes back to:

It is the product that must adapt to the client, not the other way around, or else the client will not buy the product.

Which is always true. But tailors don't work as cheaply as salespeople. So your job really is to make it seem like the salespeople are good enough. Which is how you end up with buttoned on sleeves.

The way you avoid that is you take pride in your work, set up a stylish design that allows for easy custom tailoring, and refer requests for 3rd arm holes to the competition.

Answer 2

Candied orange's answer makes a good job in explaining the economical motivation behind configurable systems in a quite metaphorical way. Still the main software engineering question remains, how we can avoid such configurations becoming overly complex.

Unfortunately, there is no easy answer to this.

Finding the right amount of configurability to make a system satisfy several different needs without falling into to trap of the inner platform effect is actually what separates good software architects from excellent ones. One has to find a balance here.

There are, however, some strategies which can help you keeping a system configurable on one hand, but still controllable with reasonable knowledge and effort. A few things which come into my mind here are:

• Orthogonality: when you add new configuration options, try to make them as independent from each other as possible, so changing one does not affect the others. That keeps the system testable and maintainable, even when the number of switches start to grow.

• YAGNI - which means in this context: don't make things more configurable than necessary "just in case". Wait until you get a real requirement to change some feature or process or label for two or three different customers, only then start to add a switch or configurable rule or configurable text to the software. When you need three different rules for a specific use case, you may implement them as with the strategy pattern and a simple three-valued configuration switch between the 3 strategies. When you need three hundred different rules, then a rule engine is probably justified.

• modularity: try to break down your system into individual small modules each with its own set of configuration parameters. Several small modules, each one with a configuration of small/mediocre size is much better to handle than one large monolith with a huge parameter list. Keep orthogonality in mind: one module which produces data under a certain configuration should pass the configuration parameters to the next module as part of its output, so there will be always a "single source of truth".

• usability: when you cannot avoid configurability, at least make dealing with complex configurations as painless as possible. A well designed configuration editor can help a lot, with a self-explanatory GUI, well written documentation direct shown near to each parameter, some useful example or default configurations and in-built tests against inconsistencies. That can avoid the need for a team of specialist, a few "power users" might be enough to handle this.

• traceability: this is for finding and repairing bugs or issues in the configuration. Care for making it easy to trace some observable effect back to its roots (which might be a configuration parameter). This is achieved by writing log files, validation of the consistency of certain config parameters and displaying useful error message.

Let me add that the idea of orthogonality (or at least some misunderstanding about it) is sometimes what can lead to overly complex configuration systems, at least when it is not limited by YAGNI. Each new customer requires a few new bells and whistles to be added to the configuration. Often, when devs notice a specific type of config requirement, they think "hey, this switch contains internally 5 other things too, which could be changed individually. So just in case one of the next customers wants them different, don't let us build 1 new switch, but 5 into the config.". I recommend to be very hesitant with this approach - your team will have enough effort to maintain the real requirements which are justified by existing customers, there is no need to add "virtual requirements" to the list. In rare cases, however, a more general and uniform solution can makes things easier to maintain - and that's where a lot experience and common sense is required.

"Compile time constants" is usually not a solution. That's a way of configuring a library written by one software vendor for other software developers. The latter can almost every time replaced by a runtime parameter. But for runtime parameters, it is usually a lot simpler to shift the responsibility for configuring between

• the end user

• the end users administrator

• some "configuration specialist" or staff member of your company

• the testers, or the automated test itself

• the developers

whilst for compile time constants, only the last group can change the switch.

Answer 3

A solution we've been using for a couple of decades now is to have complex ERP software with standard off-the-shelf configurations. The customer gets a lot less choice taking a pre-configured solution, but it is at least proven to work for similar customers.

For example an aerospace & defence customer will require serial numbers, traceability, ability to build and design complex products, project management, stage billing of projects etc.

On the other hand, the same software may go into a food manufacturer. Needs are very different here where e.g. despite being a manufacturer you need the ability to take items apart (such as a chicken to get two breasts and two legs), or you may need differing recipes per manufacturing run. Every car needs one engine, but a vat of soup may need more or less water or sugar adding depending on how dry and sweet this particular batch of ingredients was.

So these requirements are common to many customers in a similar industry so you take a pre-configured version of the software and it'll do 80% of what they need out of the box.

While this is a good solution, inexperienced customers will not appreciate the principle of KEEP IT SIMPLE, STUPID (and inexperienced implementation consultants may not enforce the necessary discipline) so they try to make the software meet 100% of their requirements, committing to endless configuration for little gain, and inevitably hamstringing the project. Better to get 80% of your needs over the line in a phase one then get that stabilised and move forwards with incremental improvements.

It is obvious that the software must adapt to the client, not the other way around.

No, absolutely not. Nothing of the sort is obvious. In many cases, software has been designed to underpin a manufacturing or business process which is proven to be efficient, and to work, by experts with way more experience and expertise in the business process than the customer. In many cases, software can be an effective mechanism for driving change within the customer to adopt best-practice processes.

In summary, many successful products have a couple of layers - the core software, which is constantly updated and improved, and a configuration layer. Then pre-configured templates are applied at the configuration layer to meet specific common needs, facilitated by the flexibility of the client to adopt best-practice processes.

Answer 4

Maybe not...

This approach, from my experience, has important flaws.

• From the programmers' point of view, the product, in its entirety, is incoherent, too big, too complex and frankly weird. It is hard to understand it and develop it.

You're judging the whole approach in principle based on how you've seen it done badly in some cases. I have no way of judging how badly designed those cases were and if there were obvious improvements that would mitigate your conclusion.

• As mentioned before, the configuration is really very complex. Being able to understand it and to use it properly is a no small feat, which is why there must be employees who specialize in this very task. [..]

Needing experts for a complex job is not inherently bad, it just should be reserved for cases where that complexity is justifiable. Depending on the degree of customization that your company sells as part of its product, this can be well justified or a needless obstacle. I cannot judge this universally and neither should you.

[..] The configuration is also dangerous, since any mistake in clicking the numerous checkboxes, selecting proper values from numerous lists, sometimes even entering pieces of code, etc, etc will result in the whole software working incorrectly, maybe even causing data corruption.

What you're missing here is a testing and validation process, rather than a need to redesign the solution from the ground up (again, I'm not saying I know, I'm saying that you're trying to blanket judge based on specific instances).

If these codebases are distinct per customer, and therefore the "configuration" is development work, this work should be verified using a testing suite. If the customer wants the solution to reticulate the splines but only if they're not part of the dinglebop, and your developers have to configure the solution to do this, then your developers should also be writing a test to confirm that the solution reticulates the splines that are not part of the dinglebop.

If the generic product is a SaaS and the configuration is done via the application's runtime (as opposed to pre-compile time like the previous example) then your developers are really just the end users of that application's configuration panels.
The solution here is to apply validation to your admin panels in the same way that you would for "normal" end users. If the devs are entering a configuration that is contradictory, then report that back to them, either as an error or a warning (depends on the context).

---

..but maybe yes?

That being said, this is what I call the spork principle (I coined it myself, I doubt it makes sense on its own).

The core issue here is that on the face of it, a spork is better than having a separate spoon and fork. Most people who see a spork for the first time find it an interesting solution. And yet hardly anyone ever actually uses a spork, instead favoring discrete utensils.

There's one common exception to this: campers and trekkers. For these people, volume and weight comes at a premium, and being able to condense two piece of cutlery into one is just one of those things that positively contributes to reducing the bulk of what they're carrying.

Back on topic, the main thing to point out here is that by trying to make a generalized tool, it start getting clumsier, and you really need to maintain awareness that the reusability is netting you more than the added clumsiness is making it undesirable to use.

But this is highly contextual. Campers and trekkers have a different set of priorities than your average middle-class household, and therefore they will balance the pros and the cons differently, leading to different decisions on whether they get adequate value from using a spork.

That's not a question I can answer for you, and it's not a judgment that I (nor you) can make universally for every company and dev team.

Answer 5

There are a fair number of companies that sell software that, in general, must be adapted or highly customized per client. SAP comes to mind as a fairly common case.

All software is additive and lives above something else. ERP software lives on top of a browser or runtime, which lives on top of a kernel. A company may choose to build their ERP software, "from scratch," but unless they're building a custom OS for it, they're just using others' code.

The same company may use an off-the-shelf ERP solution if it's configurable enough to meet their needs. There are options between the two ends of the spectrum as well.

The big difference between one end and the other is price, how much it will cost to implement an approach. Note that this cost is total cost of ownership, including maintenance, essential upgrades, etc. Therefore the company's decision would be based on what they could afford.

Since the question is about a developer's choice as to what to build: a product shop has to decide what their customers need. If they make this choice incorrectly, they will ultimately fail to sell enough of their software to survive. It is therefore in their interest to cater to their clients' needs.

Answer 6

Why do companies making software sold to other companies tend to attempt to make the code general enough to meet the needs of all clients at the same time, while adapting the product to the needs of each client through configuration?

Because the vast majority of the hard work (and lead time) of analysis and development is embedded in these configuration capabilities of the application.

Although the amount of configuration per client can seem considerable, it is often modest (and completed far more quickly and predictably) compared to the overall time and labour power invested in the development of the configurable software itself.

Once the major up-front investment has been made in creating the configurable application, clients can get a working system in place more quickly than if they had to wait for a complete bespoke development, and more cheaply than if they were to hire and mature an internal staff function capable of handling that bespoke analysis and development to a reasonable standard.

The configuration step may also use workers who do not themselves need to be adequate developers with a full range of competencies (unlike those responsible for the software development proper), therefore reducing wages, moderating competition for the limited pool of real developers in the economy, and allowing the work done by the real developers to be spread across many more clients.

These are essentially the reasons for the configurable software model.

Is there any better way than that to let the product adapt to the clients?

Better in what sense? Certainly nothing can beat bespoke development done and kept to a good standard. But the costs can be considerable, especially once good developers aren't seen as a bottomless pool but something employers would actually have to compete for.

Would stripping configuration to the bare minimum and adapting the product to each client in the code help here? Would it even be feasible?

It wouldn't be feasible because you'd run out of developers in the economy, if you needed a developer for every single adjustment currently offered by "configuration".

An alternative of course is to force clients fully into the Procrustean bed of standard software.

But even across very similar businesses in the economy, there are always differences of scale and sophistication which mean untailorable software would either be incapable of coping with the detail of bigger operations, or would be excessively complex and difficult to harness for smaller operations.

Businesses also change over time, so occasionally need to be able to grow gracefully without frequent revolutions of their software and stored data, and the application needs to be able to evolve its mode of operation like this, switching on more complex aspects in a way that continues to makes sense of existing data and is free of dysfunction.

This is not to say we live in the best of all possible worlds, but there is a lot of essential complexity in business information systems which is still not a settled science.

Answer 7

This is a subject that is big enough to fill a book, there is no simple answer. The solution depends highly on the number of customers, the overlap in functionality per customer, and the type of relationship between the product development company and the customers.

But we can limit the discussion to a company that has only a few customers, and has a close relationship with those customers, and ignore most of complexities and the business related aspects. In such a situation, it is common to give each customer their own version of the product.

The main components of the product will be the following:

1. a generic core of functionality that is used by all customers

2. internal configuration settings per customer. This is configuration for the generic core that is specific per customer. As the supplier maintains these settings for each customer, the product delivered to the customer only needs to be tested with those configurations.

3. internal specific functional modules specific to each customer. Software modules that are specific for a customer, used only by that customer, and maintained by the supplier. Make sure to keep those software modules specific to the customer, likely maintained by a customer specific maintenance team. When similar functionality is needed between 2 or more customers, then it is time to refactor the functionality, to make it more generic, and bring it into the generic core.

4. external configuration settings that the customer can change. As it is not known what combination of settings the customer is actually using, the testing of these needs to take all combinations into account, leading to large testing efforts. So try to keep these settings limited, and try to keep them orthogonal, such that the settings don't influence each other.

5. external "business rules" type of functionality that the customer can change. This is the ultimate of flexibility, and requires a well-defined, well-documented, and well-tested API. Whole business models are built around this by platform suppliers such as SAP.

The generic core needs to maintained in a generic manner, especially through testing of all possible combinations of configuration and usage. The internal customer specific functionality only needs to cater for what that customer requires, and therefore a much lower testing effort suffices.

To find the right balance between these parts is not an easy task. Especially the identification of common functionality that can be brought into the core requires a dedicated architect team who have knowledge of how each customer is using the system, and what customer specific functionality is required.

But given these parts, an architect has a way to balance different needs in a more flexible way. If the external configuration gets too complex, it might be better to encapsulate various configuration patterns into one configuration parameter that chooses which pattern to use. That brings them into the internal customer settings, where it is easier to maintain. If they get very complex, then it might even be better to make customer specific functional modules out of them.

The usage of customer specific compiler configuration for the generic core is not recommended. It makes it impossible to test the core separately from the customisations, and could even lead to modules that do drastically different tasks dependent on the settings. Instead of that, separate the generic functionality from the customer specific one, and bring that into the core, possibly with internal run-time configuration settings with customer specific values.