Visitor provides a way to treat all 'visited' (element) variants transparently in the same way, and the mechanism by which this is achieved requires all of the 'visited' classes to be aware of the abstract Visitor (or the Visitor interface). As with all of the patterns, there's room for variation, but this is a key aspect that enables the pattern to work and be utilized by client code as intended.
Besides being a way to implement double dispatch, Visitor is essentially a way to implement an abstract data type (ADT) in an OO language that happens to lack other, more convenient mechanisms.
Here, I'm using the term ADT in its strict sense. It could be described like this: you have a finite number of distinct data structures (that you expect you won't have to change or extend often), but you want to abstract that fact away and let clients treat them all as a single data type. What's needed is something to represent the type itself (in the Visitor pattern, that is the role of the abstract Element), and a bunch of operations on that type that can internally distinguish the underlying concrete data representations (in Visitor, operations are represented by various concrete Visitors).
It helps if you mentally translate
The idea is that client code has an abstract reference to an element (doesn't know its concrete type), and operates on it by selecting and/or instantiating a concrete visitor (passing operation parameters to the constructor). So, in a way, the whole elements + visitors construct is best thought of as representing one thing, one type and its operations. Every element ('visited') class is aware of the abstract Visitor type only, but that's done in order to provide implementation dispatch (the capability to call the correct method) on the received visitor instance (often, this relies on polymorphism + built-in overload resolution capabilities).
ADTs have complementary extensibility characteristics compared to objects of OOP. It's easy to add a new operation (just create a new Visitor derivative), but it's hard to add new representations (if you add a new element type, you have to update every visitor). Whereas in OOP, once you have several derivatives in use, it's hard to extend the interface with new operations, but it's easy to create new representations (you simply create a new derivative).
The different concrete data types relate to the same abstract concept, and are generally designed to work together in some way, and may have a recursive structure. So it might not make too much sense to apply visitors (operations) to arbitrary 3rd-party classes. You might try and create a wrapper, though, but it gets convoluted - and the Visitor pattern is already pretty convoluted to begin with.
You might also try and keep the ADT semantics, but break away from the Visitor pattern, by taking the dispatching responsibility out of the element; e.g., instead of
you could have
do might simply type-check and downcast). Or you might move dispatching into the operations themselves. Note that, in order for client code to use them abstractly, you still need a way to represent all element types, including the 3rd-party ones, as a single abstract type (that is, a way to assign them interchangeably to variables of the same single type).
One thing that Visitor has going for it is that it uses the compiler as a bookkeeping device - for each new derived visitor, inheriting the Visitor interface forces you to implement the operation for all supported concrete element types (whereas if you are type-checking "manually", you have to be careful not to forget some). Also, since operations are represented by object instances (concrete visitors), you can pass them around and do other interesting things with them, but that's less of an advantage since today most OOP languages support closures + lambdas / delegates, and some have first-class functions.