This specific case is addressed in the C4 Model Frequently Asked Questions. Both options - showing service A publishing a message to the broker and the broker sending the message to service B is just as valid as omitting the broker and using notions (textual notes, colors, line style, arrow heads, etc.) to show the communication.
Since both options are valid, I would look at the clarity of communicating the system to consumers of the model. My preference would lean toward including the message broker as a component on the diagram, however I would omit it if including it and all the lines necessary to represent connections made the diagram harder to read.
Depending on clarity, you could omit the broker on the container diagram, but include it on the relevant component diagrams, or vice versa. However, you may not be creating component diagrams if they do not add value. You could also consider showing the broker on a supplemental diagram, such as a deployment diagram.