If you have a use case where you never have to deal with negative numbers inside of any big integer calculation, then you could use the signed big integers implementation as well, and your performance guess is what I would expect, too, the performance differences would most probably be negligible.
If you have a use case where negative intermediate results can occur, but you want them to "wraparound" to some positive value, then in a big integer context it is not inherently clear what the result should be. For example, the outcome of "4 -5" in a 32 bit unsigned integer context is typically "2^32-1" (as long as your environment does not throw an underflow error). What should it be in a context where there is no such upper limit of 32 for the number of bits?
That makes it hard to create a useful specification of how general purpose unsigned big integer should work, assumed one wants to make this a universally useful too.
Of course, this problem could be solved by introducing an artificial, configurable "maximum bitsize" into the unsigned big integer implementation, and I would be astonished if not someone somewhere in the past had implemented such a thing. However, I think authors of widely used general purpose libraries or languages will think twice if implementing such a specific requirement is worth the hassle and if there will be enough applications to justify the additional development and testing effort.