Could someone write an opensource version of the Windows API?
Yes, but you are asking the wrong question!
Okay, after that teaser ;-) let's step back a bit.
The answer to the question "can someone write X" is almost always "Yes". Unless it is "No". Well, okay, that's not terribly helpful. Obviously, the answer to a Yes/No question is either Yes or No.
What I mean by that: there are some fundamental limits to computability. The most famous one is Alan Turing's Halting Problem. We have a rather simple mathematical function on the natural numbers. It's not quite as simple as addition, but pretty close: H(p, i) = 1 IFF program p halts on input i, 0 otherwise. That's clearly a function, and it's clearly within the natural numbers (we know that any sort of input, as well as programs themselves can be encoded as natural numbers, in fact, our modern computers show that they can actually be encoded using just the numbers 0 and 1). Yet, as Alan Turing proved, it is impossible to write a program which computes this function. Rice's Theorem is in a similar vein. There are lots of other similar results, most of which reduce or can be proven with one of those: Dead Code Elimination (impossible, equivalent to the HP), Class Hierarchy Analysis (HP), Escape Analysis (HP), and so on.
Basically, whenever you want to write a program to figure out something non-trivial about a program without running the program, that's impossible.
So, for those types of questions, the answer to "can someone write X" is always No. And that's a mathematical "No", not a "we could theoretically do it, but it's too hard/expensive/slow etc.". It's simply plain impossible. Period.
For pretty much all other questions the answer to "can someone write X" is always "Yes". In particular, in this case, you are not trying to statically analyze a program, so after what I said above, the answer should be "Yes, you can."
What you want to build could be called an abstraction layer, emulator, virtual machine (in the theoretic sense, not the "Java" sense), interpreter, adapter, translation layer, compatibility layer, or something like that (depending on how you look at it, and how you actually go about implementing it), and it is perfectly possible to do that.
In fact, all of this goes again back to Alan Turing, Alonzo Church and the very beginnings of computing. Alan Turing proved that there exist Universal Turing Machines that can take as an input the description of any Turing Machine and perform the computation of that Turing Machine. This is essentially the very first instance of an interpreter and the very first instance of a stored program computer, both before the very first actual interpreters and computers were built. Later, when the question arose whether Turing's Universal Machine or Church's λ-calculus were more powerful, less powerful, or equally powerful, it was proven that a Universal Turing Machine can compute anything λ-calculus can, by simply emulating λ-calculus, and λ-calculus can compute anything a UTM can, by emulating a UTM. The same thing was proven about other models of computation, such as μ-recursive functions, the SK combinator calculus, and so on.
In fact, for every model of computation (ignoring models that are purposely restricted, such as finite state machines) found so far, we have found that it can emulate all other models and can be emulated by all other models. This has led to the famous Church-Turing-Thesis: all models of computation are equally powerful. This includes even such physically complex things as Quantum Computers, and even physically impossible things such as Non-Deterministic Turing Machines.
So, what does all of this have to do with your question? Well, think of it this way: the Win32 API is kind-of like an abstract machine (again, in the more abstract sense, not in the "bytecode interpreter loop with garbage collection"). The POSIX API is also such a machine. And, as we have seen in history over and over again, one machine can emulate another machine.
This means that the answer to your question is "Yes, it's possible".
In fact, if you think about it, there already are different implementations of the Win32 API on different platforms: on 16-Bit DOS (Windows 3.x), 32-Bit Windows (Windows 9x/ME), and Windows NT. We can argue that WoW64 is a fourth implementation. In fact, we can argue that WoW64 is two implementations, since WoW64 on AMD64 works very different from WoW64 on IA-64.
However, like I said at the beginning: it's the wrong question!
The question you really need to ask is "How hard is it?" And I mean that both in the complexity-theoretic sense of the word and how much work you need to put into it, i.e. how complex the mapping between the two APIs is.
[Interestingly, this also applies the other way around to the "impossible" programs I mentioned above. Clearly, compilers do perform all kinds of static analysis on programs, including Dead Code Analysis or Escape Analysis. It's just that it is impossible to always do this. IOW, while there are infinitely many programs for which you can clearly determine whether something is Dead Code or not, there are also infinitely programs for which you simply cannot decide algorithmically one way or the other, and thus you have to include that code because otherwise the program might break.]
Again, let's look at the examples from above: yes, a Universal Turing Machine can emulate a Non-Deterministic Turing Machine, but the emulation overhead is exponential(!!!). And a Quantum Computer cannot compute more things than a Classical Computer, but it can compute some things much faster.
For the "mapping complexity", let's take C++ and C as examples: C can compute anything C++ can, and C++ can be mapped to the constructs of C. However, sometimes, that mapping is very direct (non-overladed C++ functions can be encoded directly as C functions), sometimes there is a little complexity involved (overladed C++ functions have to be encoded as multiple differently-named C functions, aka name mangling), and sometimes the mapping is completely non-obvious (e.g. the mapping of objects, classes, virtual member functions, and inheritance into C structs, vtables and dispatch functions).
That is, for example, the problem with Wine: they emphatically do not want to be an emulator (it's even in their name, Wine is not (an) Emulator), they want to rather provide a fairly simple 1:1 API translation layer. And as it turns out, this is possible for a large portion of the API, but impossible for some parts of the API that just cannot easily be mapped to POSIX. That does not mean that they cannot be mapped at all, it just means that within the design constraints the Wine team has set themselves, as well as the financial, resource and manpower constraints that are imposed on them by this pesky thing we call "the real world", there is no easy way to do it.
If you wanted to go about it a different way, i.e. allowing more complex translations and biting the bullet to do some emulation as well, you could go much further than Wine. The Mono team, for example, has implemented some .NET APIs in a portable fashion despite the fact that they are actually pretty tightly coupled to Win32. OTOH, they opted to not implement some other APIs because of their tight coupling (e.g. WinForms).
The way that this problem is typically approached is from both ends: translate the stuff that is easy to translate, re-write the stuff that isn't. There are some companies that have "ported" their applications to Linux by targeting Wine, and simultaneously joining the development of Wine and contributing patches to translate more functionality and at the same time re-writing those parts of their applications that use functionality that cannot be translated easily.
So, to answer your question again:
Could someone write an opensource version of the Windows API?
Yes, but the more complete and faithful you are going to make it, it's either going to be slower or more complex. In particular, if you try to map it to another high-level API that makes different trade-offs than Win32. (Otherwise you end up with something like ReactOS which maps the Win32 API to nothing more than a naked CPU, which makes it fast, because you don't have to have complex mappings in place, but much more code to implement, because you don't get to use the simple mappings.)
