If I got this right, as soon as you replace the nuget reference from B to A by a project reference, there is no project left which consumes A as a nuget package, right?
So if there is no consumer any more, it is pretty obvious that the nuget package is superfluous.
Let me scetch one additional scenario here: what if there will be another external project C in the future which wants to consume A by a nuget package - should B then be changed to consume A as a nuget package as well? Or is it ok to let B reference A directly, using a simpler project reference?
To my experience, as long as B wants to consume always the newest version of A, and A & B are part of the same product, the sheer existence of C does not justify a change of the reference mechanism. It can be perfectly sensible to let A expose itself "internally" by one mechanism, and externally by another.
However, if B is mainly a test project, written for making sure the externally exposed project A works as intended, then it is probably a good idea to let B use the exact same reference mechanism as an external consumer C. Tests for components should simulate the aspects of a real consumer as similar as possible.
Moreover, if C also references B, and you are start deploying A, B, and C as one product, it is probably a good idea not to mix the different reference mechanisms. Otherwise you end up with C referencing version 1.0 of A directly, and version 1.1 indirectly through B, which will end up in some kind of DLL hell. For this case, however, one may consider however to have A, B and C part of the same solution and use only project references.