Is this a common problem?
I read from my eventstore and use rabbitmq to dispatch the events to my listeners/projectors
There's a trap here - if your projection requires processing events in the correct order, then you need some mechanism for ensuring that happens.
The most straight forward approach is to read sequences of events from the event store, and then process those sequences, instead of trying to take messages delivered to you by a bus and figure out how to order them.
Put another way, you use the bus to alert the projector to the fact that there is more work to do (reducing latency) but still let the book of record serve histories to the projections that need it.
See Greg Young's 2014 talk on Polygot Data.
Should I ditch RabbitMQ?
Not necessarily; but you want to be careful about how you use it.
If each event has its own handler thread (meaning that events can be processed concurrently), and they are going to be writing to the same database entities, then you've got a race condition that you aren't going to be happy with.
Any given "truth" in your system really wants to be single writer.
So the middleware bus delivers events to "the" mailbox of the single writer, and it processes the messages in sequence, and you are good to go.
Now, if your system is designed in a such a way that rabbit guarantees the order of delivery, then you are in good shape.
You're also fine to use rabbit for those cases where you don't care about the order of the messages.
But if you care about the order of messages, and the system design doesn't allow rabbit to guarantee delivery order, you need that order maintained elsewhere.
Typically, this means that when you save the result of the command you are also appending domain events to an ordered history. The events are persisted into the ordered history before they are published into rabbit.
Readers that need events in the correct order use the queue to tell them that the ordered history has changed, but when they need order events they query the ordered history rather than trying to create a copy from the events that they have seen.
In effect, the arrival of the event from rabbit tells the writer that the ordered history (may have) changed, and that the writer should refresh its local copy from the source and then take any appropriate action.