There are a couple of things which make backwards compatibility non-trivial.
First, one constraint that is often used is that your code will compile without change in a backwards compatible change while in a non-backwards compatible change it may require code changes for your code to compile against the new version. Of course, this constraint isn't useful for JavaScript.
This leads to the general idea, which is that the public "contract" of the code doesn't change. Types are a formalization of an approximation of the public contract. A test suite is another formalization of an approximation of the public contract. That is, in a backwards compatible change, all tests in the old test suite should still pass. Unfortunately, what's part of the public contract is almost never fully formally specified and is often not even fully specified by any means. The public contract is what the library developers are committed to maintaining and the only thing you can rely upon as a consumer, but, especially in a language like JavaScript, there are myriads of details that are also visible which aren't part of the public contract and thus can't be relied upon. Of course, this doesn't stop people from relying on them.
Finally, there's the awkward case where a library doesn't meet its own public contract, i.e. it's buggy. In this case, consumers obviously are forced to rely on the actual behavior of the library and not its intended behavior. Fixing the bug, bringing the library more in line with its public contract, will now break those consumers. Avoiding this is called maintaining bug-compatibility and is a nightmare most developers want to avoid. I would say the "unspoken consensus" is that bugs can be fixed without notice.
Ultimately, it's hard to make any significant change to a code base (as opposed to a pure addition), especially in a language like JavaScript, that produces no programmatically observable difference. What serves as the (partial, informal) specification of the public contract usually leaves it unclear what behavior is guaranteed in various corner cases. Even given a complete specification, it is usually prohibitively expensive to actually verify that the code meets it. So at the end of the day, what constitutes "backwards compatible" is usually not sharply delineated, and even if it was (in a meaningful sense) it would probably be too expensive to actually verify.
So, yeah, the developers hope that the change is "backwards compatible", just like they hope that the code behaves as its nominally specified to do, but everyone has been around the block enough times to know that such hopes are unfounded.