"Open source" typically includes that you provide the source code of your program in an understandable, compilable and changeable form. So any kind of artificial licence check could be easily disabled by anyone who is able to compile your code and has a basic knowledge of programming .
Of course, you could use a license which forbids to change anything in your program (or at least to distribute any changed branch of your software), so people are only allowed to read your program, not to change it. What you cannot enforce, however, that people will remove your license key check secretly, or that people will do this outside of your jurisdiction. And when your licensing scheme is completely open, you cannot hinder people to imitate that licensing scheme, generate their own keys or provide a fake license server, even without changing the original code. (In fact, this is done regularly even when programs are closed source, but for an open source program you make this "approach" more feasible to the "average developer", not only to people with experience in reengineering.)
So one of the most popular ideas to get paid for open source is to have an important part of the software not beeing included in the source code, the part you provide from your server. The more functionality you put on the server side, the lesser is the risk having the server become rebuilt by anyone else. A server which does provide only a boolean information if a "certain license key is valid" is far too easy to be copied. In fact that means that your monetising model becomes a service model. For example, as a vendor of a multiplayer game, the game client might be open sourced, but the server controls lethal parts of the game, and you can charge people for using that service for a particular duration.
The other option you have for monetising is to create an open source library which can be used freely by anyone for non-commercial software, freely for inhouse-software, but paid for commercial, off-the-shelve software. People will pay you because there is a certain risk that a licence breach can be proven by you, because they get paid for their software (so they could affort it to spend part of that money to you), and because they want to get updates of your library from you. For such a distribution model, however, you typically don't need any license keys.
EDIT: Of course, there are more alternatives to get money for open source software than the two above. This Wikipedia article lists a few.
