I know you can't take someone else's code and change it from GPL to MIT, but if you are the only contributor to a repository can you freely switch it from GPL to MIT?
If you published code once under GPL, that version of your code is GPL code, that is nothing you can change again. Someone else (in fact, thousands of people) could have made a copy, that copy is under GPL, the GPL text is part of the code, and you cannot enforce anybody to change his copies "just because you wish to".
However, if you create and publish a new version of the code, you can change your license to whatever your want for that newer version. That is true even if the only effective code change is the license itself. IANAL, but as long as you are the one and only copyright holder (which I assume, since you wrote you are the only contributor), you can do with your code and the usage rights you grant to others whatever you want.