This is precisely what a Contributor License Agreement is for. Do not, under any circumstances, assume anything about what rights someone else does or does not waive. You're in doubt, so talk with your actual contributors, not random strangers who don't even know what project you're talking about.
The MIT License is not much of a license at all. All you're saying is "use this if you want, but don't take my name off it and don't sue me if it blows up in your face." The MIT license is not in any way a sticky copyleft, and the negotiations about as to who gets to claim copyright at all are entirely beyond the scope of the license.
What you need to decide, and possibly discuss with your contributors, is who owns the copyright to the code. Do you want to own it jointly with them, or do you want to own it free and clear? If Facebook comes to you and offers sixteen billion dollars to hand over the copyright, do you want to share the money? If you decide to switch licenses to the GPL and give your code to the FSF, would your contributors want a say in the matter?
Do not assume. Decide what you want, write it down, and either give your past contributors copyright or refuse their pull requests until they agree to your new CLA.