I really don't mean to attack other answers, but is nobody else writing automated tests here?
Here's a fun reading from Martin Fowler for anyone doing Scrum without proper software engineering practices. Robert C. Martin also says a lot about this here.
So, to my answer... In short, it goes like this:
Yes, "randomly" refactoring code is allowed in Scrum, as long as the team decides that it should be done. (After all, it is self-organizing)
And now for the long answer:
It's evident that leaving more and more technical debt after each Sprint is a recipe for disaster. Soon, everyone will slow down as the code bocomes more messy; every change will be harder to make because the code is so tangled and messy that it takes longer to find the spots to change than to make the actual change. It gets even worse if you have to make a change in a big and messy module that you know nothing about, it becomes impossible to gain/keep productivity when adding/switching people in the project, and so on.
If a team wants to keep its velocity steady, they must be able to keep the code base clean in order to continuously increment the software. Refactoring is a mandatory practice if you want to keep your velocity throughout the project's life cycle, and if you want to mitigate the risk of adding/switching people on the project, and if you want to be able to make changes in modules you know nothing about, and so on.
However, refactoring is a very dangerous activity. I repeat - it is a very dangerous activity. That is, unless you have enough test coverage to be able to safely and freely change the code base. If you can just hit a button to check if nothing broke, refactoring becomes a very safe activity; so safe, in fact, that it is part of the cycle of TDD, which is the practice that allows you to create such a test suite in the first place.
But, as the teams in Scrum are self organizing, in the end your team must decide what is the right thing to do. I hope to have given you some arguments in the case you have to convince anyone. (Give special attention to the links in the first paragraph, and every other article they point to)