When you write unit tests for A, you mock X. In other words, while unit testing A, you set (postulate) the behaviour of X's mock to be X1. Time goes by, people do use your system, needs change, X evolves: you modify X to show behaviour X2. Obviously, unit tests for X will fail and you will need to adapt them.
Woah, wait a moment. The implications of the tests for X failing are too important to gloss over like that.
If changing the implementation of X from X1 to X2 breaks the unit tests for X, that indicates that you've made a backwards incompatible change to the contract X.
X2 isn't an X, in the Liskov sense, so you should be thinking about other ways of meeting the needs of your stake holders (like introducing a new specification Y, that is implemented by X2).
For deeper insights, see Pieter Hinjens: The End of Software Versions, or Rich Hickey Simple Made Easy.
From the perspective of A, there is a precondition that the collaborator respects the contract X. And your observation is effectively that the isolated test for A doesn't give you any assurance that A recognizes collaborators that violate the X contract.
Review Integrated Tests are a Scam; in high level, you are expected to have as many isolated tests as you need to ensure that X2 implements the contract X correctly, and as many isolated tests as you need to ensure that A does the right thing given interesting responses from an X, and some smaller number of integrated tests to ensure that X2 and A agree on what X means.
You will sometimes see this distinction expressed as solitary tests vs
sociable tests; see Jay Fields Working Effectively with Unit Tests.
Shouldn't we focus more on integration testing?
Again, see integrated tests are a scam - Rainsberger describes in detail a positive feedback loop that is common (in his experiences) to projects that are relying upon integrated (note spelling) tests. In summary, without the isolated/solitary tests applying pressure to the design, the quality degrades, leading to more mistakes and more integrated tests....
You will also need (some) integration tests. In addition to the complexity introduced by multiple modules, executing these tests tends to have more drag than the isolated tests; it's more efficient to be iterating on very fast checks when work is in progress, saving the additional checks for when you think you are "done".