Really, it depends.
To take your XML example, consider how much refactoring might need to happen if you needed to extend or change your code to use object(s) represented by a different format; for example, wouldn't it be nicer if your code didn't care whether it was dealing with JSON, XML, Protobufs, etc?
On the other hand, you have to weigh the cost (in up-front development time, and in creating an additional layer of indirection) of writing an additional layer of abstraction compared with the potential benefit and likelihood of needing to support other data formats.
Also, consider unit testing - for example, how easy is it to use the library with unit tests? In the case of XML, it's generally easy to write some mock XML data, or provide a mock XML file. In the more general case, you might consider writing an abstraction layer if the library is likely to interfere with testing (for example, if it requires a remote endpoint or hardware device to work correctly).
If there's no significant impact on testing, and you can't foresee any need to add that kind of indirection, then consider the YAGNI princple, but also aim to at least structure your code in such a way whereby you can minimise the impact of future refactoring in case you need a new layer of abstraction later on.
Ultimately it comes down to up-front cost vs expected future cost. There are really two scenarios:
The cost of writing more complex code up-front involves spending longer writing that code, and may involve future developers having a harder time learning how the code works, but it may also save buckets of time implementing future requirements.
The up-front cost of throwing code together as quickly as possible can be heavily outweighed by the need to repay significant Technical Debt in future, and in the extreme case, risks creating a Big Ball of Mud.
While you shouldn't fall into the trap of writing elaborate solutions to simple problems, you should always be looking for ways to minimise technical debt; you can often achieve this by investing time creating automated tests. If you're struggling to decide, perhaps a more useful way to look at the conundrum is whether creating a new layer of abstraction will reduce the time it takes to write those tests.