The main issue with your takeaway is that you are using your entities as your domain objects.
Now, to be fair, I actually take that approach in several projects (built on Jason Taylor's Clean Architecture framework, which also does this out of the box), but this comes at the inherent cost of letting EF into your domain.
The predominant advice you're going to find online is to not do that and to keep your datalayer away from your domain (as I understand it, this idea is what your concern is based on). I personally think there are valid use cases to do so, but let's table that discussion for another day. Both approaches work, and you have to pick the approach that is the most appropriate for your use case.
In reality, you have two choices. Either keep your entities as your domain objects and thus accept that your domain will contain an EF dependency, or separate your entities (infrastructure) from your domain objects (domain), so that all EF dependencies are kept in the infrastructure layer and not the domain.
In either case, your concern ceases to exist, either because it becomes a non-issue or because you've separated the datalayer from the domain.