"I think this is a good solution, but I do not recognize the pattern"
Your diagram does not depict the pattern/structure you're asking about in a particularly explicit way, so I'll continue based on this:
"MQTTMessenger calls a on_message method, which uses a reference to
the VirtualDevice's update_props method"
&
"when the device state properties are changed it [... makes use of...]
a reference to the MQTTMessenger object's update_state method"
This is the same as having an interface that one class accepts (or maintains a reference to), and the other implements, it's just that you are constrained to a single method per interface. What corresponds to the interface here is the signature of the method. So this is, broadly speaking, just a way to do dependency inversion: two concretions - your VirtualDevice and MqttMessenger classes, rely on an abstraction - the (signature + meaning) of the method each of them references.
So, you have this
and this
The arrow with the white triangle indicates "implements", the other one is "references" (either as an internal field, or as a method parameter). When put together, you get:
The Instrument class then assembles these together; for the classes to remain decoupled, it is important that none of them depend on Instrument. You should be able to write an executable test for them without using or mentioning Instrument at all - the test code itself should be able to take over the role of a part assembler.
You may have seen a similar structure in some other contexts; e.g. Robert Martin's blogpost on Clean Architecture has a diagram with an an inset depicting an example of a dependency that crosses an architectural boundary:
This is almost exactly the same thing, except that the Presenter and the Controller are merged into a single object, and that in your case, the context is not the UI/Application boundary, but a communication layer / device layer boundary.
In this diagram, the layers are color-coded; note that the interfaces (boxes with an <I>, the ports) are "owned" by the Interactor (both of them are in the same layer as it), and that all dependencies (arrows) crossing the layer boundary (going from one color to another) are pointing in the same direction).
This arrangement determines which of the layers knows about the other layer, and sort of constrains how you put them together. Here, the Interactor and the associated interfaces are deemed more important than the presentation (UI) layer, so the UI depends on the application logic.
Now, this image might not exactly parallel your layer (or component) structure; in terms of where you draw your architectural boundary, you have two choices.
If the core/heart/soul of your application is about the devices themselves (if that's where the core logic lies), then you'd do this. Here, it's all about the devices, and the Messenger conforms to what they dictate. This favors being able to plug in different kinds of messengers (or other kinds of delivery mechanisms), assuming the update_state & update_props methods are robust enough for the needs of the device-side logic (meaning, you rarely need to change the signature of these methods when you need to plug in a new kind of messenger).
On the other hand, if core/heart/soul of your application is about the communication protocols between the devices, you'd probably place the boundary here:
This makes it all about the communication infrastructure, with the devices being able to plug into it by conforming to the interfaces (here, methods) required by it. This favors being able to plug in different kinds of devices, assuming the update_state & update_props methods are robust enough for the needs of the communication protocol (you rarely need to change the signature of these methods when you need to plug in a new kind of device).
So this has some implications on what these methods are actually going to look like, and how they'll evolve as your application matures.
Now, at this point, this boundary feels predominantly conceptual, but another way it comes into play is if you want to divide your application into separate packages, making the boundary physical. Currently, these interfaces (method signatures) are implemented by a method on one of your primary classes, but you could in principle provide a standalone function or even a specialized object (all variations on a similar theme, in the spirit of the Command Pattern Robert Harvey mentioned in a comment). Such a free function / closure / dedicated class would be placed into the same package as the "interface" if the "implements" arrow doesn't cross the boundary, and into the other package if it does.
Aside: In statically typed languages, one package (or DLL, or whatever) would actually statically reference the types in the other.
