It's important to note that this scenario can happen in both legal and illegal forms.
This can occur legally if the company owns or has access to the copyright associated with the code. For example, smaller, independent development teams may wish to sell or license their copyright to a larger firm. Likewise, if the bulk of the code's copyright can be acquired, the "unattainable" portion may simply be thrown out and rewritten.
On the other side of the coin, the code may simply be taken illegally. New Corp, Inc. may not care what the legal ramifications are. The project may be too old or abandoned or New Corp thinks they'll win in a lawsuit. Or New Corp may be registered in an area where this sort of "acquisition" simply isn't defined as illegal. Not all OSS licenses are enforceable, especially not the ones that were cobbled together by non-experts in the law. Not all project owners may have the ability to enforce their copyright claims, and larger organizations such as the FSF may not have standing in that jurisdiction for that claim. TL;DR - this area can get hazy and ugly really fast.
Transition & prevention
How does the transition happen and what can be done to prevent it beyond choosing a difference license?
The transition occurs when New Corp, Inc. acquires the code base and places that copy under their control. New Corp's developers then begin working on their version of the code base making whatever changes the corporate overlords have decreed necessary. The actual mechanics of this fork will vary based upon the repository. And while it's a philosophically big deal, it's really pretty underwhelming in practice. get all from OpenRepos and then checkin to the PrivateRepos.
What can be done to prevent it short of never distributing the source? Nothing. Sorry.
Let's use GPL (GNU public license) as an example. GPL requires the source to be available to anyone receiving a legitimate copy of the project. There are no provisions that allow the holder of the source to refuse delivery of the source to a legitimate holder of a copy of the GPL'd application. It goes against the grain of Free software, and that's why the GPL copyleft is in place.
Potentially, you have after-the-fact legal courses of action to pursue. But those are all after the source has escaped and been forked, not before. And in some jurisdictions you won't have any legal recourse. And all of this assumes you're even aware that the fork occurred. You may never find out.
Ethics
What are the (ethical or social) responsibilities for the company? (For example: Giving back to the open source project would be the ethical thing to do)
Ethics are local to the culture. So temper this section with that grain of salt. A full discussion of cultural consideration of ethics is beyond the scope of this answer and off-topic for Programmers.
I have noticed that the programming community tends to react negatively to a hostile fork. Heck, in some cases the same community still reacts negatively to a friendly and legal fork. It's a pretty complex community.
From an FOSS perspective, there is an expectation that New Corp is going to "repay" the community for the contribution it forked. The terms, conditions, and duration of that repayment are as varied as the number of OSS projects in existence. Some in the community (think Richard Stallman) will never be content with an open project going closed. Others will look for the benefit provided to the community at large and will judge based upon that. And others simply won't care because they never knew of or cared about the origin project.
Source availability
If the open source version and closed source version are both available, how does the competition affect either product?
It really depends upon how comparable the two code bases are with respect to functionality, performance, and stability.
If the code bases remain similar and New Corp is friendly to the OSS community, they may contribute their updates back into the base project. In this case, everyone benefits. It's not a "competition" in this case, but more of a mutually beneficial collaboration.
If the code bases wildly diverge and New Corp is not friendly to the OSS community, then there still isn't a competition. The more feature rich product survives and the less rich product tends to die off. Note that this can go either way - the closed version could die off if the open source version continues to innovate or better meet the needs of the community.
The reality will be somewhere between those two ends of the spectrum.
Example
Red Hat has two main distributions - Enterprise Linux and Fedora. EL is their "closed" licensed version and Fedora is their community edition. Due to the GPL, much, if not all, of the EL edition is released in source form. Another project not affiliated with Red Hat called CentOS picks up the changes to EL and distributes that project after some minor rebranding.
There were some grumblings when Red Hat forked into two separate editions, but by and large, it's been a pretty workable agreement. The Fedora community wanted features rolled into the distribution faster than what Red Hat's enterprise customers were comfortable with. Enhancements to the code bases flow in both directions.
