The GPL writes:
You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions:
So this condition only applies if your work is "based on" the library, which the licence defines as follows:
To “modify” a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a “modified version” of the earlier work or a work “based on” the earlier work.
That is, your program is "based on" the library if and only if it is a derivative work according to copyright law. The legal definition of that term varies somewhat among jurisdictions, and usually does not directly address software. For instance, the US Copyright Act writes:
A “derivative work” is a work based upon one or more preexisting works, such as a translation, musical arrangement, dramatization, fictionalization, motion picture version, sound recording, art reproduction, abridgment, condensation, or any other form in which a work may be recast, transformed, or adapted. A work consisting of editorial revisions, annotations, elaborations, or other modifications which, as a whole, represent an original work of authorship, is a “derivative work”.
What this means for software must be interpreted by the courts, based on earlier similar rulings. I am not sufficiently familiar with the relevant case law in your jurisdiction to say with certainty how a court would decide your case. One could argue that "replacing the GPL library with own code" is an act of translation, particularly if your code is strongly inspired by the GPL implementation. Even reusing the API of the GPL library could land you in hot water (see Oracle vs. Google).
If the answer matters to you, I recommend seeking competent legal advice rather than asking strangers on the internet.