ABI and API are different, but related things. An API is a contract that describes functionality. The ABI describes how modules communicate at the machine code level. The two can change independently, and one doesn't (or shouldn't) affect the other.
The ABI describes things like how parameters are passed to functions (on the stack, in registers, etc.), how system calls are made, how shared data is accessed, etc. An ABI is machine specific, operating system specific, often language specific, and also application specific.
An API, on the other hand, is often machine, operating system, and language agnostic.
There are large parts of the ABI that you don't often have to think about, because your language compiler and linker handle them for you. You might have to worry about it in some cases, for example the difference between
stdcall in C and C++. Staying with the same ABI for years is pretty easy for these parts. For example, code written for Win32 in 1995 will still work today. Even libraries that were compiled 20 years ago can be linked with 32-bit code that was compiled today.
Whatever API you're using should be steady across versions. Vendors make a concerted effort to keep the APIs backward compatible so that code written to use version 1 of the library can still link with version 5. Or 17. Granted, the API might be extended, but functions that were available in the first version usually exist and work the same many versions later. The internal implementation might be (very often is) different, but as long as the functions meet the initial contract, it doesn't matter how they're implemented.
Now, where you can get into trouble is if you change a data structure that's passed to or returned by an API function. As was pointed out in the comments, a change to a data structure that doesn't require a source code change, is considered an ABI change. That could be something as simple as changing structure packing, or rearranging the order of the individual structure members. Not to mention adding a field. In any of those cases, the size of the data structure could change, but the new version might very well link with an object module that was compiled for the older version. That would definitely break things.
Those are breaking changes that, although they don't require any source code changes, do require a recompile. And in my opinion if it requires a recompile, then it should be considered an API change.