Any straight answer is going to be extreme. Clearly there are cases in which the deadline is so tight that you must use ugly code, and there are cases where the code is so ugly that it's worth missing the deadline to improve it. What you need is methods to judge which you're in, and perhaps methods to set realistic deadlines that allow time to write better code.
Don't save the cleanup for later. Unless you habitually have periods with nothing to do but refactor, there is no "later" in which it will somehow become higher priority to tidy up the code than it is right now. The routine is "red, green, refactor", not "red, green, do something completely different for two weeks, refactor". Realistically you will not change the code until next time you're revisiting it for some other reason, and you'll probably be on a deadline then too. Your real options are to fix it now or leave it.
Of course well-styled code is better than badly-styled code, assuming you plan to ever read it again. If you plan never to read it again, then do not tidy it up. Ship the first thing that passes the tests. But that's a pretty rare scenario, for most programmers it happens approximately never. Ignoring that case, only you have the details of your real case to make a judgement how much it costs to fix vs. how much it costs (in increased future maintenance) to not fix it.
There are certain things that are no harder to fix at the point where the code requires maintenance, than they are to fix now. These don't actually benefit you much to fix now. The most obvious are trivial to fix (whitespace errors and the like) and so it's difficult to imagine that you have time to ask this question but not to fix them ;-) For that ones that are not trivial and are of this kind then OK, you have some code that isn't ideal but you must be pragmatic. It works and you're on a deadline. Use it.
There are certain things that are considerably easier to fix now than they will be later when (a) they're not so fresh in everyone's minds; (b) other things have been written that rely on them or imitate them. These are much more valuable to fix now, so prioritise them. If you don't have time in your deadlines to fix these, then you need to push as hard as you can for longer deadlines, because you are building up debt in your code base that you'll probably have to pay next time you visit the code.
The preferred method of fixing code is through a review process. Comment on the problems you have with it, and send it back to the junior to change. You might give examples of what you mean and leave the junior to find all the cases in the code that they apply to, but don't just finish their code for them. If you do then you give them no means to improve.
You should write common problems up into a style guide that says "don't do this, do this instead", and explains why. Ultimately the reason is allowed to be, "in order to make our code aesthetically consistent", but if you aren't prepared to write down your rules with some justification then you probably shouldn't be enforcing them either. Just leave each programmer free to choose.
Finally, beware of the tendency to tweak stuff indefinitely. The returns diminish, and you need to learn through experience where they're still good. It is absolutely essential that you form a realistic idea of what is good enough, or else you can't have that negotiation in which you make sure that your deadlines give you time to create "good enough" code. Spend your time on things that are not good enough.
