As @Sean mentions, refactoring should be separated from feature additions to keep things really clean.
I think it is as useful to review refactoring as it is to review feature additions.
Any given refactoring needs to be explainable, if it is sufficiently simple, then in terms of the small refactoring steps, but more importantly whether small or large, in terms of the better domain-applicable abstraction that is being provided.
Good abstraction operates in terms of consumer-appropriate concepts and behaviors that work together to support the consumer's usage scenarios.
In an abstraction that is no longer providing for newer usage scenarios, sometimes the incompleteness is made up by adding code to the consumer often that knows too much about the implementation details of the abstraction and needs to reach around to a lower level to make up the incompleteness. This creates technical debt by spreading responsibilities between a provider and its consumer.
This situation indicates refactoring to create improved abstraction. Such a refactoring may augment an existing set of concepts and behaviors, or may replace existing abstractions with a new set of higher-level more domain-appropriate concepts and behaviors.
When we do refactoring, it should be explainable: the new abstraction incorporates another capability or another concept with capabilities, or perhaps it is a whole new set of simpler or more useful concepts & behaviors.
In either case, it should make things easier for the consumer, so the next level up can "stand on the shoulders" of the refactored abstraction instead of working around it.
In some cases, however, the implementation burden for a given abstraction is too high (it cannot be provide it without serious compromises) and in those cases the abstraction should provide something lower level, simplifying the implementation and being presented to the consumer in a different way.
Refactoring is about negotiating the boundaries of our abstractions, and, these adjustments in boundaries should be explainable to those reviewing code.