When we work with legacy code and need to do changes, we first write tests on the current behavior. That way we can implement new changes with confidence. We can even refactor the code.
That may be reflecting sometimes your working process, but in my experience, a way more efficient process is:
you write tests
you refactor to make a change easier
you implement the change
This way, it becomes more apparent that you refactor when there is a real reason for a change, not just because the code "is not clean any more".
Now try to apply the same measures to your tests: you don't refactor your tests because "they are not clean any more". You refactor them when they start hindering you to make easy changes to your existing code.
For example, when you have ten tests all calling the same public method of a class under tests, whilst in your production code that public method is only called in one place, then this is a form of code duplication by tests which may hinder you to change the signature of that public method.
I would usually leave it that way unless you really get the requirement for the latter, or more general: when you notice this code duplication requires you to make the same change to your tests in several places.