"It's a completely rewritten version of another one [...]"
Strictly speaking, something completely rewritten isn't considered a version; at least not in computing where we take "version" somewhat formally. For example, it is misleading an incorrect to call Linux a "version of Unix". The word widely accepted for this is "clone".
Something rewritten also cannot possibly be a fork, even if it resembles the original to the point of being a clone (such as by having similar or identical interfaces).
Since you've tried modifying the original and it didn't work, and then embarked on a rewrite, that bridge has already been burned: the rewrite isn't a version, and so it cannot be considered a fork. It isn't even a clone, because the interfaces are different!
It's a new library which covers a similar area of functionality as the old.
If the library internals are complicated and worth reusing, and the new interfaces can be easily adapted, to use the existing internals, then a case can be made for extending the library: keep the old interfaces in place (perhaps providing a way to remove them at compile time to save space). If that approach was feasible for this library, then you wouldn't be doing a complete rewrite, though.