When your gut is telling you that you should probably do some refactoring, It's likely it's your instincts telling you a little late that you've been putting something important off for too long.
I understand "code-smells", red-green-refactor and other thoughts, but often I feel that the best time to refactor is not the first time you write the code, but the second or third time you are using the code and realize that it is actually a problem and is in actual use.
There are effectively two levels to refactoring. The first is the obvious issues that appear when you first code. These are the little optimizations that cost you very little to do up front. Things like keeping your methods and classes small, and adhering to DRY and SRP. Then you have the additional stage of dealing with major flaws in your design, which may not be immediately apparent until your code has a couple of miles under it. It's this second level that you are talking about, and yet in order to ensure that later refactoring isn't too costly, you need to have already written your code in such a way that the effort you later envisage is made easier and less costly, which means doing an early refactoring.
As Jeff mentioned in his answer, "time is money", particularly in companies where the workload is high and the risks even higher. Time spent up front making sure the code is in its best possible state is time saved later, when teasing out what should have been an easy refactoring turns out to be a major operation.
When writing software, every moment spent improving your code up front is time saved later, when you're really going to need it. The earlier you refactor, the clearer your later changes will be. It's like making a down payment in today's dollars against future technical debt which will be in tomorrows inflated dollars.
In any case, refactoring shouldn't be a task that you put off until some mysterious future when the software is already complete and stable, as it increases your risks later when the stakes are much higher and the product much more difficult to change. Refactoring should be a part of your daily activities, and this is the essence of the Red-Green-Refactor philosophy that you mentioned.