Is it possible for the author to register a logo or the name of his/her application when it is open-source because it uses a gpl library (for example)? The application uses the library but it has its own features, that is it's not a modification of the library. So everyone can see the source code, downloading an "anonymous" non-branded version from Sourceforge, but none can use the logo or the name and so the author can sell it from his/her web-site (for example) to non-expert users, only after payment. Is he/she obliged to give directly the source code, or it is enough that there is the non-branded version available on Sourceforge or other official repository?
This question, apart from the fact it's hardly related to programming, suffers from a major issue. Which country are you considering? In most of Europe in general, and in France in particular, your intellectual property is protected, and a logo, as it is an artwork, will be protected without any other step unless you expressly make it public domain.
However, other countries and regions may differ, and you would need a lawyer with expertise in global intellectual property to answer decisively.
Generally speaking, the way to protect a logo or a name of a product is to register it as a trademark. The details of how you register a trademark, and the protections that this gives you will vary depending on your country. Talk to an IP lawyer for more information ... or check with the agency of your government that is responsible for trademarks.
Open source licenses are based on copyright, and this is generally speaking orthogonal to trademarks and patents. However, a genuine open source license would allow anyone to remove your branding / logos / etc, and distribute a no-name version or a version with their logo and branding. (Some open source licenses require some kind of acknowledgement of the origins of the code.)
So basically, what you are wanting to do is possible from a legal perspective (IANAL, etc).
However, it is not clear that this will achieve what you want. It will allow you to sell your branded version (subject to the conditions of any upstream licensing of course) but it won't prevent someone else from stripping out your brand, etc. And the chances are that someone will do this ... assuming that your product is worthwhile in the first place.
So really, branding per se doesn't offer you any competitive edge with an open source product. You should really be focusing on offering value for the customer's money in another area than old-style licensing fees. For instance, try to make your money on offering support services, customizations, and things like that. Or by getting sponsorship.
And once your brand has an association in the customer's mind with good value for money ... then the brand starts to have a value.
This is not to say that trademarks are not without value to you. For a start, they can defend you against some opportunist scum making trouble by trademarking your product's name and using that as leverage to extract money.
If you are distributing binaries of a GPLed program YOU must make the source code available. Pointing to a third party site is not enough.
To address the question of trademarks in GPLed code. If the code is GPL then you must distribute the same source (Trademarks and all) used to build the binary.
As an example see Red Hat Enterprise Linux, the source with trademarks is made available by Red Hat. CentOS and Scientific Linux remove the trademarks when they repackage RHEL.