The critical distinction is between:

 1. The optimized code is as simple or simpler than the un-optimized.

 2. The optimized code is more complex (and therefore more error prone and harder to modify in the future) than the un-optimized code.

In the first case, sure go ahead. In the second case you have to weigh the investment in development time (including opportunity cost on not using the same time to fix bugs or deliver features) and the future higher cost of maintenance for a more complex solution. You have to weigh this cost against the observable improvements to performance. How will you perform this judgement if you have no idea what the performance cost is? A function might be obviously inefficient, but if it only takes a few milliseconds anyway, a 1000x performance optimization will not provide any value.

Second, your intuition about performance might very well be wrong - and you will never know if you "optimize" before measuring. For example many developers tend to think that say a O(log n) algorithm is faster than a O(n). But you don't know that. The O(n) algorithm might be faster as long as n is below some threshold. What is that threshold? You probably don't know. And what is n actually in your particular program? Is it usually above or below this threshold? How will you find out? 

The really difficult decision is when you can go down different roads in the architecture. Should the network communication be JSON over HTTP(simple) or protocol buffers over TCP (performant)? The problem here is you have to decide up front, before you can even measure performance, if you don't want to waste work by having to change protocol later. In this case you cannot just start out with the simple version and then optimize later when it turns out to be a problem. But you shouldn't just choose the most performant version by default either. You will have to do some educated guesses and projections.

---

I note you state that profiling "as often as not" gives the same result as you intuition based on your understanding of the program. I take that to mean that you have a 50% success rate in predicting the best way to allocate resources for optimizing. Given the short and long term cost of misapplied optimization, that is not really a very good rate to rely on.