Hi I'm new to programming, and I've always wondered what would be the point of contributing open-sourced codes if there is no way to ensure everybody who uses the code will adhere to the licenses (or is there?).
For many people, it's vanity, getting your name associated with something awesome (that's why small OSS projects tend to stay small, and large ones tend to have to have very rigorous processes to vet potential new submitters, they simply get flooded with applications from everyone and their dog).
For others, it's an attempt to get others to do their coding for them. Throw a half baked idea out onto sourceforge (for example), hype about it using a few accounts with other names on slashdot (for example), and sit back and wait for the code to come in (which usually fails, but people still keep trying it after all these years).
Then there's people (and usually companies) who've something they developed in house for internal use and think would be useful for the world at large but they don't have the time, money, people, whatever to polish it up for sale and/or to market it (maybe it's something that doesn't fit into their existing product portfolio well for example). They put it out there as open source, using a foundation or other legal entity created for the purpose as a front to own and oversee development, stocked with initially their own people, and make it known (good marketing) that they're sponsoring that foundation (by hosting its website and allowing their employees to spend some of their work hours on it).
And then there's the real altruists, the people with their heads stuck in rose coloured clouds if you wish, who do it out of a sense of "everything should be free" (without considering that they can only contribute to those projects because they have a job doing exactly the same thing for pay).
Those together are probably 90%+ of contributors to open source projects. Legal issues like others using some of that code in their closed source projects don't matter to most of them, at least no more than as a nagging feeling somewhere. And it's not really much of an issue. If you see most open source licenses there's no restriction on commercial use of the open source library or product as part of something else. Many ask that you credit them if you do, but not even all of them.
And I seriously doubt the situation you describe comes up all that much. Most open source projects are not very suitable for it, either being tightly grouped libraries or complete products (and most of those not very mature, though the best ones are very good at all), so the effort needed to extract bits and pieces useful for your own use and splicing those into your product is likely to be higher than the effort needed to create the functionality from scratch (maybe using the OSS source as inspiration or guidance).
Don't assume that it's not possible to detect that a compiled binary is a copy of someone else's code.
For a start, all string literals go through the compilation process unchanged. If a closed-source program has lots of string literals that match those of an open-source project, then it's bound to raise suspicions.
After that, there will be someone, somewhere, who is obsessive enough to decompile/reverse-engineer the code to work out that it's doing the same things as the open source library.
If the open-source code is from a major project, then the next thing is that you get a letter from their lawyers, suggesting that you might like to open source the project that contains the code you've copied. If you fail to do that, the next step is a lawsuit. During the discovery phase, they will ask you to present to the court your source code, to prove that it isn't just a copy of theirs.
If you lose the court case, you could be looking at damage payments, and being forced to withdraw your product from the market.
I think the way you are phrasing the question mixes together two different things: why you would contribute to an open source project and how an open source license can be enforced when binaries are distributed. The point of contributing to an open source project is to contribute to the project and people do it for all kinds of reasons. The second question is the more interesting one.
At that point different things may happen. Most larger projects have actual teams or individual people who take on the job of investigating licensing issues or an individual developer may have to just manage this herself. Usually the way the process would work is just like with any other civil legal issue. First there is a lawyer-approved or drafted letter sent to the authors of the suspicious code from the author or project. Then there is a lawyer-approved letter that responds. That can resolve the whole thing sometimes. For example, maybe they say "We clean room reverse engineered your code and here is our documentation of that." or "No, our code actually does something different or has a different signature" or "It has come to our attention that, unknown to us, a contractor has used code that is similar to yours. Our company would never knowingly use code contrary to its license, and this was specified in the contract. We have removed it and released an update to all customers, and that contractor no longer works with us." Then the original developer and the lawyer decide how to respond. And so on. In rare cases they may proceed to an actual court filing and have a discovery phase. In a discovery phase a judge may require that the source code be shared.
A lot of times with GPL violations the violation is failure to notify customers that they have a right to a copy of the code, adding an inappropriate EULA, failure to include a copy of the license, or distributing outdated versions when customers ask for the copies.
Many permissive open source licenses require that the original licensing information be included with the code, so it is not just GPL that has these kinds of issues.
Contributing on open source projects is, at least in my opinion, a very idealistic thing to do. Idealists thend to believe in the good in people - and that means, that other people don't steal source code to copy-paste it in their closed source projects.
On the other hand, music, movies, books can also be easily copied - and that has not (yet) stopped artists from publishing new stuff.
For some licenses you are permitted to release closed source binaries. These licenses are known as "permissive". Proponents of these licenses believe that downstream users will contribute back appropriate bug fixes or features because it is a more efficient way of maintaining the code. That is the incentive to share is economic and process driven.
Another category of license (weak copyleft) allows binary distribution as ling as there are no modifications in that code. This is most commonly used for libraries. In this case there is the same economic and process driver but it is backed by a legal requirement to release the source code of modifications.
The final category is copyleft which uses the law to require all source to be released if it includes any copyleft code. The economic and process drivers still exist here but proponents of copyleft (and weak copyleft) tend to believe this is not enough without the law to add teeth to the argument.
Open source does not necessarily mean that the code can't be taken into closed source. There are two kinds of schools of thoughts on this: Some licenses like the BSD and MIT license family allow that given some attribution is being made. And this is used - Microsoft for instance used BSD's TCP/IP stack for a long time in Windows (Win 95 till XP, I think). The reasons there are that people think it is great that their code runs in so many places and are happy to make the world better. It allows helps to ensure that TP/IP implementations of different systems are compatible. Part of the thought there is that even if some closed source product includes this code the open version will be more attractive. In this side creators and users are quite equal and have both quite somefreedom in usin the code.
The other school of thought is around the Free Software Foundation's GPL license. Here the thought is that ensuring "freedom" of the software is more important than ubiquitous availability. The freedom here is that the code will always be available to everybody and thus the user can always fix bugs and continue using it even if the vendor goes down.
Now in both cases taking the code into other software can be illegal (with BSD missing attribution etc, with GPL not making the software GPL etc.). This might go unseen often but binary analysis or even test of functionality can prove that the code was included and legal action can be taken. http://www.gpl-violations.org/ documents cases where this was the case and provides legal assistance.