Application code should only be as good as necessary, but library code should be as good as possible, since you never know how your library is going to be used. So when you're writing library code, it needs tooto be good in all aspects, be it performance, robustness, or any other category.
Also, you need to think about performance when you design your application and when you pick algorithms. If it isn't designed to be performant, no degree of hackery can make it performant afterwards and no micro-optimizations will outweigh a superior algorithm.