Is this last assumption right?
Yes, what else do you think?
Within DDD there is the tendency where Entity equality is based on having the same id not the same memory address.
This has nothing to do with DDD. "Database objects" always need a unique key or Id, there is typically no "memory address" to be stored in the DB to distinguish them. So there is always an id available, which can be also used for checking equality for objects in memory, regardless from using the identity map pattern or not. Having an "Identity Map" in place, and assumed in your context you can make 100% sure that no object can be created without checking the "Identity Map" first, it is indeed possible to replace the id test by a memory address test, but in practice the advantages are IMHO very small.
I always had the "feeling" of having a unique instance of my objects*
Honestly, having a "feeling" about something like this is IMHO a different word for not knowing how something works. And yes, using a framework without knowing what goes on is always "dangerous" (I prefer the term "error-prone"). To be more specfic: when using an ORM, one should inform himself thoroughly if the ORM already provides something like an "Identity Map" or "Object Cache", and what features it provides exactly.
Should I normally be concerned about my objects having multiple instances?
This depends heavily on the kind application you are going to write, on the kind of business transactions, on the number of records involved, on the kind of queries, and so on. It is perfectly possible to write applications which create objects from the database in a local scope, manipulate them and write them back, without caring about other parts of the application which might have a 2nd copy of the same "db object" at the same time. But if you need the same piece of information in your application always in sync, this might not be the way to go.
Or even being out of sync with the database?
Again, this depends. For lots of applications you cannot even avoid things to get out of sync, at least for a short time (think of a scenario where a second client changes the DB over the network, without any "push" mechanic). But how long this is tolerable, and if certain transactions need to be 100% sure to have the newest version data, depends on your requirements.