I always start the day thinking "I'll easily get this done by the end of the day" and set what looks like a realistic target.
So why do I never hit it? The task always ends up taking 3x longer due to unforeseen bugs, last minute changes etc.
Is it just me? I don't seem to be getting any better predicting what can be done in a day.
Because you were never taught how to plan.
Planning is a skill, just like coding or writing. But somehow it is left out of nearly every curriculum.
It needs to be learned and practiced, and your estimates of your own capabilities need to be continuously updated. This is why work practices like Agile emphasize measuring your past actual work and comparing it to your estimates, so that you can improve your planning abilities.
As others have said, you need to account not only for the task, but for all its predecessors, attendant tasks (e.g., learning how to do X), and you need to be cognizant of your own internal mental biases that will prevent you from correctly accounting for the way you actually do work.
Train on it, and who knows, you might get better.
Hard to believe that nobody has mentioned Hofstadter's law yet.
I think the real answer is that your planning always assumes a best-case scenario, as if everything worked immediately an no interruption ever occured. In real life, you start coding, then the telefone rings, you are distracted for 5 minutest, spend another 15 minutes on stackoverflow or programmers.stackexchange to calm down and refocus, do some coding, run into an unexpected behaviour of some API, do some googling, spend 2 hourse to test the possible solutions etc.
In other word: "best-case" only happens in your dreams.
Every programmer, once in a great while, has a perfect day. You wake up 5 minutes before your alarm goes off feeling great. Breakfast is made and on the counter, along with fresh coffee, so you can grab something and head out the door. During your commute you hit every green light, and traffic seems to be especially light. Contemplating the day ahead of you, you are able to fully understand the design and consequences of the task ahead of you, which has been well planned with firm requirements.
You get to work and you find that you have no important emails, no voicemails waiting, and your co-workers are either out or in meetings that you don't have to attend. You fire up your editor and are immediately in the zone, you can feel the structure of the code and see your data structures and algorithms fitting into place within a beautiful and cohesive whole. Thoughts flow through your hands to the keyboard, entering perfectly formed code that is elegant, maintainable, and with not a bug to be found.
During the day you work with no interruptions, the office is quiet and you are so focused that you're never tempted to spend any time catching up on the news, blogs, etc. When you compile and run your tests, you find that everything works without a hitch, of course you knew it would, and at the end of the day you commit with no conflicts. Glancing at the clock on your way out you realize you put in 12 hours and it felt like a brief 20 minute coding session.
That day, that perfect day, is what we assume we'll have every time we have to estimate anything.
Don't forget about meetings, people interrupting you, etc. Unforseen bugs are difficult to predict, but over time you should be able to get an idea of how many bugs you uncover within a certain time frame. When estimating how long something will take, you have to consider the context. I.e. "Assuming I don't get interrupted or uncover bugs I should be able to do something in X amount of time"
As a little exercise for yourself, consider doing the following:
You'll find some patterns start to emerge, and can plan accordingly for those. Any time you tell your manager an estimated time of completion, just caveat it with the assumption in the first paragraph. You might be surprised how accurate your estimate was when you remove the time spent on interruptions and bugs.
When you are working to a bug list or feature list, you are probably already doing the first and third bullet point. That little exercise will tell you where all your time is going, and you might be surprised at the answer.
You may want to expand you timeframe of predictability. Can you determine what you can get done in a week? If every task is taking three times longer than you thought, it sounds like you're consistent enough to be predictable. You just need to adjust by 3x ;)
cuz you simply ignored the fact that unforeseen bugs may occur.
Do some statistics on the average time you spent on the bugs, and take those time into account when you make your plan.
Because you aren't planning correctly. Ouch.
I bet if you keep a running total of how much you slip by (on paper even), then adjust your estimates up by that %, you'll be able to plan correctly.
FWIW, software is notoriously hard to estimate. McConnell (of Code Complete fame) has a book out on it even.
Something I often find myself doing is getting distracted by random stuff that is unrelated to what I'm doing. A todo list can help with that; when you think of something, write it down and do it after you finish what's in front of you.
Because I spend too much time on Stack Overflow.
Urgent/Important Matrix may be worth considering to see where does your day go. Is it on urgent but not important stuff like unprepared meetings and interruptions? Is it on urgent and important stuff you didn't know at the start of the day? Just an exercise to consider where does your time go.
I'd tend to think most interesting things are important or else why are they interesting? Just a thought.
That's a good question and one that I'm constantly pondering. I tend to think that
