we can't have duplicate Category names either, that would just cause confusing and redundancy.
Assuming that you have a business invariant that requires no duplicate category names, then the responsibility for enforcing that invariant belongs in some aggregate. Since enforcing the rule requires having access to all of the category names currently in use, that same aggregate must include, as part of its state, that collection of names.
This is basic aggregate construction - the consistency check and the state go together.
where should this validation take place?
In the reserveName() command of the aggregate.
Should there be a "service" layer that takes a UI model and turns it into the domain model and attempts to validate it?
No, there shouldn't be anything unusual here. Client wants to use a new name, so it sends a representation of the
ReserveName command to the application. The application rehydrates the command, locates the appropriate command handler, and dispatches the command to the handler. The handler loads the aggregate, and invokes the reserveName() method using the arguments supplied. This exactly follows the same pattern as every other command you dispatch to the domain model.
If the aggregate discovers that the category name is already in use, then it throws a domain exception, and your application reports it to the client; again, exactly like every other command routed to the domain model.
Which aggregate is "the aggregate"? Beats me -- you need to look in the ubiquitous language to discover the aggregate that enforces this invariant. For instance, there might be an aggregate that has this responsibility and nothing else. In a toy example like this one, you might pretend that CategoryNameReservationBook is something that makes sense to the business. Or perhaps NameReservationBook, with one NRB instance being used for category names, and another NRB instance used for blog post titles, or some such thing.
When you discuss this with your domain experts, you may find that "no duplicate categories" isn't really a hard constraint. In ddd, it is important to hear the difference between "that must never happen" and "that shouldn't happen"; the latter phrasing implies that it does happen occasionally, and if so there's likely to already be a process for that contingency.
You may also find, in the discussion, that this isn't a business constraint at all; in which cast you get it out of the domain model completely -- if it's something real, maybe the application handles is alone, or maybe there is some other domain model (perhaps in another bounded context) that is responsible for it.
Trying to support this constraint with services and the like doesn't really work; the service running in this transaction can't see what's being changed in other transactions, so you get a data race. You have the same problem trying to use a specification -- the data used to initialize the specification is stale, and another transaction can be running here.
Domain models with race conditions are inherently suspicious anyway; if two users working on separate tasks are blocking each other, then something is wrong. Udi Dahan went so far as to write that race conditions don't exist -- that in the business world, the conflict isn't prevented, so much as it is detected and mitigated.
So check, and double check, and triple check that maybe the uniqueness constraint isn't all that, as far as the domain experts are concerned. But if it really is a business rule that there be no duplicates, then it follows, unconditionally, that there is an aggregate responsible for keeping track of that.
Upon further consideration, a uniqueness problem like this may be scoped; the programmer's blog has a category named rest, and the health blog has a category named rest, but they aren't the same thing at all. This could be supported by giving each blog its own reservation book, but it may also be a hint that Category merely an entity, subordinate to some other aggregate.