For an open source project, is it legal and acceptable to link to commercial libraries?
For example, an open source application I'm involved in would benefit from using commercial 3rd party user interface library.
Although legality might be something to consider, the bigger question here is related to practicality and usefulness (and, as you said, acceptability).
If you release your product as open source, but no one can use it without purchasing some commercial libraries, how useful is your product going to be? Unless your target audience already has this commercial library installed for some other purpose, I highly doubt that they would buy it just to use your application.
So legality aside, I don't think potential users will consider this acceptable behavior.
A lot depends on the specific license - GPL v3 in particular is very strict, whereas BSD is extremely permissive.
GPL is an interesting case. There's specific terms about binary linking intended to prevent people creating an "open source" application that's dependent on a closed plugin for essential functionality. The terms can be interpreted quite broadly, irrespective of their original intent. I've argued elsewhere, for example, that every application can be considered a plugin relative to the operating system based on the description given in the GPL FAQ (data structure sharing etc) - hosting an application on Windows might (in a paranoid frame of mind) just about be considered a violation of the terms of the GPL, even though there are many GPL applications available for Windows.
It matters a great deal what you mean by "involved in" the application. Open source licenses don't restrict the copyright holders, they restrict what others who want to copy and distribute it can do.
It also matters what the commercial license is, and whether you hold the copyright on that or not.
Bottom line, if you hold both copyrights, no problem. If you hold neither copyright, most likely a problem unless the license terms of both are very generous. If you hold only one copyright, it's going to depend on the details.