This portion of the GPLv2 seems to address your situation:
If identifiable sections of that work are not derived from the Program, and can be reasonably considered independent and separate works in themselves, then this License, and its terms, do not apply to those sections when you distribute them as separate works. But when you distribute the same sections as part of a whole which is a work based on the Program, the distribution of the whole must be on the terms of this License, whose permissions for other licensees extend to the entire whole, and thus to each and every part regardless of who wrote it.
Based on that, it would appear that the source of library A can be distributed under a BSD license, and anyone who enables a build of A+B is responsible for understanding that the resulting combined work is under the GPL.
You asked about the GPLv3 specifically. I'm less familiar with it, and its wording is less clear:
To “modify” a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a “modified version” of the earlier work or a work “based on” the earlier work...
A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an “aggregate” if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate.
It sounds as though library A is separate and independent from library B, and I don't think that optionally depending on library B would count as copying from or adapting part of library B, and assuming the intent of GPLv3 is similar to GPLv2, it again appears that the source of library A can be distributed under a BSD license, and anyone who enables a build of A+B is responsible for understanding that the resulting combined work is under the GPL.
I am not a lawyer, this is not legal advice, etc.
Examples of existing libraries:
- PoDoFo, a C++ PDF library, is licensed under the LGPL, but it depends on FreeType, which is dual licensed under the GPL and its own attribution-required BSDish license (both of which would offer additional restrictions beyond the GPLLGPL). PoDoFo's licensing page only mentions its own LPGL; it makes no reference to its dependent libraries' requirements.
- The Chromium Embedded Framework (CEF) has multiple dependencies, with license ranging from BSD to LGPL to "other," but its own code license is simply listed as "New BSD License."