Application code should only be as good as necessary, but ___library code___ should be as good as possible, since you never know how your library is going to be used. So when you're writing library code, it needs to be good in all aspects, be it performance, robustness, or any other category. 

Also, you need to think about performance when you ___design your application___ and when you ___pick algorithms___. If it isn't designed to be performant, no degree of hackery can make it performant afterwards and no micro-optimizations will outweigh a superior algorithm.