40

Today, I updated ZBateson\MailMimeParser the PHP e-mail parser library from 1.x to 2.x.

Soon enough, my PHP error log started filling up with errors.

Noting where it happened, I found out that it had to do with their ::parse(...) function: https://mail-mime-parser.org/upgrade-2.0

An additional parameter needs to be passed to Message::from() and MailMimeParser::parse() specifying whether the passed resource should be ‘attached’ and closed when the returned IMessage object is destroyed, or kept open and closed manually after the message is parsed and the returned IMessage destroyed.

That is, instead of picking one of those new "modes" by default, the author(s) simply chose to break all existing code.

Frankly, even after re-reading that page multiple times, I have no clue what the new parameter actually does. I have set it to true just to make the errors stop happening, but I'm worried that this is somehow not the right choice.

My point, and question, is: Why do library developers knowingly break existing code like this? Why not at least have it default to either true or false, whichever is the most reasonable?

Before you tell me that I should have read the upgrade instructions before updating, I sometimes do, but when your life consists of nothing but dealing with constant updates of all kinds of software, you eventually get numb to all the changes and stop spending the time and effort to do so. Is it really reasonable that updating a library (in particular) should break existing code?

And this is not some sort of edge-case of the library, either. It's literally the #1 reason for it to exist in the first place, sure to be used by every single user: parsing an e-mail blob!

15
  • 11
    I think this question should be directed at the developers of the library in question.
    – JacquesB
    Sep 2, 2021 at 18:11
  • 34
    It doesn't have to be answered specific to this library. It's a general enough question, even though it cites a narrow example. Sep 2, 2021 at 18:18
  • 8
    In this specific case, with a default option you will either have a memory leak, or silently break certain use cases. "true" is the safer option, since it will close the resource automatically. But "true" will not work if the stream is processed by other functions further down the line, without the IMessage-Object. e.g. asynchronous processing, caching and such things. In these cases you have to specify "false" and make sure the resource is closed at the right time to prevent a memory leak.
    – Falco
    Sep 3, 2021 at 11:38
  • 59
    Note that the authors of the library followed semantic versioning principles and changed their Major Version number, which generally indicates breaking changes may occur. I always consider that before upgrading any library, no matter how numb I am. Sep 3, 2021 at 14:23
  • 13
    Having breaking changes is literally the definition of a Major Version Change.
    – njzk2
    Sep 3, 2021 at 21:56

7 Answers 7

187

A major version upgrade literally means they intend to break things. You shouldn't upgrade to a new major version unless you're prepared to deal with it. Most build systems have a way to specify you're okay with automatic upgrades to minor versions, but not to major versions.

APIs break for a number of reasons. In this case, I'm guessing it's because what they would want to set the default to would be surprising to some users, either because it's not a typical convention for the language, or because of history with this library. This way, instead of half the users suddenly getting a difficult to explain "file is closed" error whose reason is difficult to find in the release notes, everyone gets a "missing parameter" error that they can easily look up the purpose of.

Remember, not everyone uses the library the same way as you. When you have a diverse user base, you have to make compromises in the API to accommodate everyone. A change that seems unnecessary to you might be just what another user has been waiting for.

7
  • 60
    Along the lines of what you've said here, the reality is that people make mistakes, including library authors. Once you've published a mistake like this in your API, it's difficult to resolve without breaking clients. A major release is when such things are addressed by responsible library authors.
    – JimmyJames
    Sep 2, 2021 at 18:59
  • 52
    Btw, "Major is going to break things" (& Minor is adding new things but backward compatible and Patch is just bugfix) is called Semantic Versioning Sep 3, 2021 at 7:42
  • 2
    What infuriates me is that they then drop the old library. "Ok, now that GTK2 is out, no need to ship GTK+ any more". Or "Now that Java16 is out, no need to ship Java15 any more" — yes, there bloody well is; our build system won't work with Java16. Sep 4, 2021 at 6:51
  • 6
    @EdwardFalk Especially for major libraries that you depend on, you are responsible for knowing their published release plans. For instance, you should know that Java has LTS versions and non-LTS versions, and non-LTS versions will not be supported after the next release. If you need LTS support, use an LTS version. Personally, I'd hate if more languages/libraries took the Python approach of "support the old version forever because people can't be asked to upgrade, splitting the community as people continue to make new code on the old version". Either upgrade, or accept you don't get updates.
    – Azuaron
    Sep 4, 2021 at 10:38
  • 2
    @EdwardFalk as long as it's OS, there is no dropping. What was made remains available and usable forever. Of course, people that put out free libraries are not obliged to keep work on some legacy version because you want new features there :)
    – Džuris
    Sep 4, 2021 at 15:49
55

My point, and question, is: Why do library developers knowingly break existing code like this? Why not at least have it default to either true or false, whichever is the most reasonable?

Because sometimes it's better to force someone to explicitly make a newly added choice, as opposed to making it for them and effectively having to guess.

If my usual restaurant tomorrow starts making two versions of the one dish that they had, I want to choose which dish I have from now on. I don't want them to choose for me and then run the risk of me getting a dish that I do not like and did not knowingly order.

And this is not some sort of edge-case of the library, either. It's literally the #1 reason for it to exist in the first place,

This argues in favor of forcing consumers to explicitly make a choice, exactly because this behavior is so essential to the library's purpose.

The fact that two approaches are now implemented suggests that there is merit to either approach, and one is not superior to the other in every possible way. If it was, then only the superior one would have been implemented.

from 1.x to 2.x

Major version updates tend to break stuff. That's why they're major version updates. They are the biggest scope on how a library can be changed.

If this happened in an update from 1.0.1 to 1.0.2, I would agree with you. Breaking existing code should only be done in a major version update.

Is it really reasonable that updating a library (in particular) should break existing code?

If all road infrastructure today still had to support horse-drawn carriages, the development of road infrastructure would be significantly hindered.

If you're never allowed to break anything, you stand in the way of innovation, and this is precisely how things (especially software) die a silent death.

20
  • 4
    @SteveChamaillard I'm not saying that consumers should be left to choose every single thing. I'm saying that when there is something the user should configure, that it can sometimes make sense to not assume a default option and instead make the user consciously configure it. And that's not even considering we're talking about a major version upgrade here, where breaking changes of this caliber are well within reason.
    – Flater
    Sep 3, 2021 at 1:36
  • 12
    @SteveChamaillard Secondly, if OP did not want to deal with the new menu with increased options, he should not have requested menu v2.0 and instead should have stuck with the old one. It's counterintuitive to ask for a (I can't stress this enough: major) update and then be upset that things have changed.
    – Flater
    Sep 3, 2021 at 1:41
  • 3
    The chef analogy fits better if the website developer is the chef, and the end users are the customers. The customers don't care which knife the chef uses, but the chef certainly does, and would likely be very annoyed if the knife manufacturer switches out his knife from under him.
    – BenM
    Sep 3, 2021 at 3:08
  • 11
    @SteveSummit "I saw a horse-drawn carriage in my city a few weeks ago." The question wasn't whether it was possible for a carriage to still go on some roads, but whether carriages were actively considered in the design of the road you saw. "In fact, nothing has been done to the road infrastructure to hinder the use of horse-drawn carriages." You're inverting the logic. I'm saying that the hindrance is to the road infrastructure, not to the carriages. e.g. hooves easily destroy "whisper asphalt" which is used nowadays to combat noise pollution.
    – Flater
    Sep 3, 2021 at 7:31
  • 6
    I think the point is that, probably, the old version was wrong (say, the dish was palatable but toxic in the long run), and there was no backward compatible solution; the best solutions required choice (say, the toxic ingredient was to be replaced by A or B, but diners have strong opinions about both A and B, or some people are allergic to A or B).
    – Pablo H
    Sep 3, 2021 at 13:46
12

Some good answers already, however, let me add my two cents from some real-world experience.

More than often, though usually acting in good faith, some API designers are pretty ignorant what kind of casding effort such decisions might cause. They have probably a wrong idea about how much client code has to be fixed by such a change and how much organizational and communication effort can be triggered by a single new mandatory attribute. Or, they have an idea, but actually no economical motivation to take care for their libraries user base.

What in a library vendor's own organization may require 10 minutes to fix, because they know their lib well and have access to the full code which relies on it, could require to hire a new developer in another organization, for example.

Over the years, I have seen many of those non-backwards compatible scenarios, and in at least 50% of them, I am sure if the designers had put a little bit more thought into backwards compatibility, they could have saved us a ton of working hours.

11
  • 8
    Assume I am a principal engineer at acme company. I develop the foo project which is very useful to my fellow acme engineers. I open source it as version 1.0. The world loves it. I make some breaking changes that make it even more useful to my employer. Why would I spend even one second thinking about backwards compatibility? I can just release it as 2.0. If you don't want breaking changes then stay on 1.0. If you want the latest and greatest then deal with the breaking changes. If you want the latest and greatest and don't want to deal with breaking changes, then recruit me.
    – emory
    Sep 4, 2021 at 0:45
  • 6
    The answer to "Why" may be that you want people to keep using your product. The response "if you don't like the changes stay on 1.0" may work for some users, but I'd start looking for a new library if I was forced to stay on an old version because the maintainers kept making changes which weren't backward compatible, but could have been with a bit of thought. In general deliberately forking your user base into version 1 and version 2 users doesn't seem sustainable. Sep 4, 2021 at 4:31
  • 1
    @emory: thanks for the agreement to my answer.
    – Doc Brown
    Sep 4, 2021 at 6:23
  • 1
    @DavidWaterworth We can partition our user base into: (1) the users who pay the bills; and (2) the free riders. If the bill paying users want the breaking changes then who cares what the free riders think. They are welcome to tag along for the ride, but they do not get control of the steering wheel.
    – emory
    Sep 4, 2021 at 16:14
  • 2
    @emory: I have seen ignorance about the consequences of missing backwards compatibility not only in open source products, but also in paid products.
    – Doc Brown
    Sep 4, 2021 at 19:07
7

Consider when you'd break things yourself. Off the top of my head, I can think of a few reasons:

  • You're updating the API to:
    • Conform to a language convention
    • Shift to a more applicable programming style (say, have a function return with partial application where it makes sense)
  • You're refactoring some implementation, and that allows for a more effective API.
  • You've updated the implementation to the point that the API is no longer helpful. These updates can be as trivial as renaming a function, which is something everyone's done at one point or another.

All of these are aimed at improving the code quality, and, when it effects you, API quality. Sometimes devs make bad decisions, and sometimes the API changes decrease the quality. But most of the time, these are incremental changes aimed at slowly and incrementally improving code: both internal code providing the API and external code relying on the API.

So what do I do?

Here are the two things to do:

  • Care about updates to the code you rely on. If you read release notes before updating (and potentially breaking), you'll be prepared for the consequences of API changes. This can be a pain, but it's ultimately effort invested in code quality.
  • Automate the above. Lots of build systems (though this is language dependent) have the ability to only update dependencies when they don't break, usually utilizing a machine-readable versioning system like Semantic Version (SemVer). SemVer is really simple: start at 1.0.0 as soon as the API is stable. Increment the third digit (i.e., 1.0.0 -> 1.0.1) on updates that don't change functionality, the second digit on updates that change the API in a backwards compatible way, and the first digit if you break the API. Then, if you rely on code that just had a major update, you'll know that you need to put some time aside to fix it.

That last bullet point is always going to be difficult (and slightly controversial), but, complementary to the first bullet point, can be exceedingly useful.

Essentially: breaking things is usually good: be prepared.

1
  • 1
    re "Conform to a language convention", I hope people ignore the C2x proposal to recommend that "size" arguments precede the pointers to the objects whose size they're describing. Yeah, it's annoying that writing functions with VLA arguments whose size follows them requires old-style function declaration syntax, but the solution to that would be to fix the rules for ANSI-syntax prototypes to say that array dimension expressions will be evaluated at the start of the function's execution and need not be resolved until then.
    – supercat
    Sep 3, 2021 at 14:57
2

Ultimately this comes down to the elusive and controversial goal of backwards compatibility.

As a developer of application A, you would like to spend your time doing three things:

  • adding features
  • fixing bugs
  • if you have done your job well, and there are no features to add or bugs to fix, bask in the glow of having written a long-term stable application that continues to be useful to its users even though you don't have to do a thing to maintain it, such that you can move on to bigger and better things.

And if every last bit of the functionality of application A is code you wrote yourself, you might have a hope of achieving this. But to do that you would probably have to reinvent lots of wheels, and that's no good, either. Another fine software engineering principle is reuse, or standing on the shoulders of others. So there is almost certainly at least one resource R that your application A depends on.

So then the question is, as the maintainer(s) of resource R go about their work, probably striving for the same three goals as you — adding features, fixing bugs, trying not to do any extra work — how much are they supposed to worry about you and application A?

You'd like them to worry about you a lot: you'd like perfect backwards compatibility; you'd like them never to make a change that breaks your application. They probably agree that this might be nice, but they probably also claim that it's not realistic: that sometimes, a bugfix or a code refactoring or an evolving need may force them to make a backwards-incompatible change. Or there may be "legacy" features which are on their way through "deprecated" and on the way to "obsolescent", which the maintainers of resource R are finding it's just way too much work to continue to support (which is of course why they're marking those features as "deprecated" or "obsolescent").

The problem is supposed to be ameliorated by a whole separate, third class of software: dependency managers D which are supposed to ease the job of managing dependencies between applications like A and resources like R. Once you've discovered that application A works with R version 1.x but not R version 2.x, and if you don't have the time or energy to rewrite A just now, you can explicitly record this dependency somewhere, and then your users will get a helpful error message telling them that they're screwed after upgrading the shared library for R on their machine.

At the end of the day it's either a tradeoff or a stalemate. The maintainers of R may try, but they're probably not going to manage (they won't have the time or energy) to achieve as high a level of backwards compatibility as the maintainers of A might like. (For any pair {A, R}, of course.) So, once in a while, the maintainers of A are almost inevitably going to be disappointed, to discover that they can't just bask in the glow of having a long-term stable application, because their application has broken, through no fault of their own.

But you have my absolute sympathy, user16508174. Dependency management can be a real nightmare (which of course is why the term dependency hell was coined), and I regularly seethe with impotent rage against it myself. I wish the maintainers of resources R worked harder to maintain backwards compatibility, or better yet, got things right the first time more often so that they didn't get stuck in these binds in the first place. But my wishing for it doesn't make it so, and I have little choice but to resign myself to the occasional (or even pretty regular) disappointment on this score.

The other thing, as you may have noticed from the tone of the answers and comments on your question, is that there may be a certain amount of religious fervor going on here. Not only are you supposed to accept that your application A is going to be broken from time to time by forces over which you have no control, you are not even supposed to complain about it. This is the way of the world. You are supposed to be glad that the maintainers of resource R have the freedom to fix bugs and add features without worrying overmuch about backwards compatibility. You are supposed to celebrate the extra time you get to spend learning how to use dependency managers D and painstakingly recording every last intricate dependency that application A might have. You should not want to bask in the glow of long-term stability; that's a confession of some kind of laziness. You are not supposed to be troubled that, after users upgrade R on their systems to v2.x in order to satisfy the dependencies of some other application B, your application A will be broken, and that those users are about to be pestering you to spend time upgrading A to use Rv2.x whether you wanted to or not. This is, again, the way of the world.

Finally, lest this answer be discounted as mere whining, let me say what I would like: I would like the maintainers of any resource R to work harder at achieving backwards compatibility. I know this is asking a lot; I know you're working hard already, and that there aren't enough hours in the day. But the reason I want this is simple: you are, presumably, maintaining resource R as a service to those who use and depend on it. You are doing your work in order to save them work. Presumably, there are more of them than there is of you. So a relatively small amount of work by you is, presumably, leveraged into a huge time savings on behalf of all your users. And making their lives easier by not forcing backwards-incompatible changes upon them is one way you can achieve this.

2
  • 3
    I think there's some truth in what you say, but the other answers have given good reasons why breaking compatibility can actually be a service to users: maintaining the old behaviour doesn't give you your ideal application, it gives you one that the library author knows is wrong, but they need your help to fix it. Just as you want library authors to make more effort to maintain compatibility, they want you to make more effort to understand the libraries and why changes need to be made.
    – IMSoP
    Sep 4, 2021 at 14:06
  • I don't think there's any "religious fervour" about this: it's straight economics. The amount of effort a supplier is prepared to make to reduce costs for their users depends on the economic leverage that the users have over the supplier. In today's climate of open source components or products that sell millions of copies for $50 each, users have very little leverage over the supplier. In the mainframe days when your top ten customers were 90% of your revenue, they had a lot more. Sep 5, 2021 at 21:40
2

One of the few things I remember from my CS undergraduate course - now 50 years ago - is David Wheeler's quote "compatibility means deliberately repeating other people's mistakes". Over time you learn that the original design was wrong. It can be wrong because it creates a security weakness, because it leads to poor performance, because it prevents you adding new features that people need, or because it creates a usability problem and a support hassle. As a library designer, you then have to make the decision whether the costs of breaking compatibility justify the benefits.

One thing I learned when I started doing open source software is that this changes the equation. When the users aren't paying you anything, you don't have the same kind of obligation towards them: you can devote your attention more to future users and less to existing users, and you can avoid the substantial costs of maintaining old interfaces that clutter the code and increase your development and support costs.

Another quote, this one unattributed: "the future is longer than the past". That says that getting it right for future users is more important than reducing the pain of upgrade for existing users.

4
  • 1
    Although I understand the mentality, and that this is the case... They may not have the same legal obligation to support existing users/code, but they still have a moral "obligation". If I knew in advance that library X is maintained by somebody who sees me as a worthless lowlife whose time and energy can be freely wasted and whose security/application stability doesn't matter just because I'm not paying them money, I would never start using that library or ever support the author in any way. Sep 5, 2021 at 21:29
  • @user16508174 you get what you pay for. If you don't want to pay, you can't expect support; OTOH you can keep using the old version as long as you want, or you could write your own implementation and use that. Sep 5, 2021 at 23:46
  • 1
    @user16508174 No, the developer of an open source package has no moral obligation to support the users of that package. I think there's a reasonable moral obligation to make the software do what it says on the tin, but there's no obligation to provide future versions or enhancements, or to make any future versions or enhancements compatible with the original. It's not that the developer sees you as worthless, it's just that by giving you a free lunch today they're not accepting an obligation to give you a free lunch every day for the next five years. Sep 6, 2021 at 11:19
  • Incidentally, commercial vendors also sometimes discontinue products. For example, Microsoft have discontinued development of .NET Framework, and told their users that the future is with .NET Core, which is not compatible. I dare say that some of Microsoft's biggest clients, like airlines and banks, have made their displeasure known. But for individual users, we have no grounds for complaint: it was never part of the contract that development would continue for ever. And if that's true of commercial software, it's certainly true where there is no contract. Sep 6, 2021 at 11:28
1

As someone who's authored dozens of libraries on Nuget and Github, I can tell you that backwards compatability is an important consideration.

Despite some other answers here, I don't believe most libraries authors deliberately set out to break backwards compatibility. Indeed, it is often possible to add many new features and improve existing ones without breaking any existing code that uses the library. And i think you'll find that most library updates do maintain backwards compatibility.

That said, there are times when a better approach is found. And the decision is made that a breaking change provides more advantages than the disadvantages of breaking that backwards compatibility.

I use C# and it allows you to mark elements as obsolete. This generates a compile warning but still compiles, allowing someone more time to refactoring their code.

2
  • I find that many users upgrade in big jumps, they go straight from version N to version N+5. This means that the strategy of marking an interface as deprecated or obsolete in one release and then withdrawing it in the next is less effective than it might be. Sep 5, 2021 at 21:35
  • @MichaelKay: Which is the reason most of my obsolete members still remain in my stuff. No one said you have to remove it in the next version. Obviously, if I ever take it out, it could break code. But this provides an approach that can lessen the impact of breaking changes. Sep 5, 2021 at 22:03

Not the answer you're looking for? Browse other questions tagged or ask your own question.