When you arrive in the morning, you find that your software does not work anymore, even though it did when you left yesterday evening.
What do you do? What do you check first? What do you do to stop being angry and start working on your problem? Do you blame your colleagues and go directly to them? What can be done to avoid being in such a situation?
The usual suspects are:
You thought it worked yesterday, but after a full day of work you were too blind to realize that it didn't work.
This morning you no longer can refer to what was in IDE cache memory yesterday.
The workstation has rebooted last night or a nightly maintenance operation cleared /tmp directories.
Something has changed in the code base: check whether someone (possibly yourself) has commited changes between your last compile of yesterday and your last compile of today.
Something has changed in the support libraries: check whether those libraries have been recompiled or upgraded. The cause may be inside the project for specific libraries or outside if a new version of an apparently independent package has been deployed.
Something has changed in the testing environment: new version of a virtual machine, a stub that has been modified, changes in a remote database server...
Something has changed in the compilation chain: changes in Makefiles, new version of IDE, of compiler, of standard libraries...
1) If it's not working today, it wasn't working yesterday either.
You thought it was working, but it wasn't.
2) There is a problem, and it must be solved.
Don't think about who's responsible for this or about blaming others.
If nothing has changed between yesterday and today (like I presume reading your question), it means you should do a better job at testing your code before actually state it's working.
To avoid this situation you have to do proper Testing and Debugging.
Define "working" and test the boundaries of your code routines.
One way of doing this is having an automated set of extensive tests run during the night, so that the day after you can check if something went wrong and fix the problems.
Trying to find someone to pass the blame is unconstructive and does not solve problems. Don't do it.
If something worked yesterday and does not work now, then either you have non-deterministic behavior (like a race condition) and having it work yesterday was just luck, or something has changed between then and now, and you need to find out what it is.
How exactly you find out which is the case and how it can be fixed depends on the specifics of the situation, but it always helps to be methodical in eliminating causes, i.e. don't change 5 things at once and stop looking if that helps - find out which specific thing caused the problem, and perhaps write down how to fix it so you can look it up when it happens again 3 weeks from now.
Using the appropriate diagnostic tools (debugger, profiler, network analysis tools) can also make a big difference.
I have worked with code that appeared to change overnight and after a while I came to the conclusion this was due to malevolent pixies crawling into my codebase at night and changing things in such a way that in spite of the fact it worked yesterday, it now doesn't work at all. Indeed in classic Schroedinbug style, not only does it not work now, it is transparently clear that there is no way that it ever could have.
Over time I have realised that it's just possible that in fact pixies have nothing to do with it and that possibly my "time to go home, that will be good enough" last build doesn't get the detailed testing and attention that perhaps it deserves.
My first assumption when I encounter this in the morning is that it is probably my fault as I'm usually the one responsible for my own features or corners of software I'm working on. My second assumption is that I might as well get that coffee now. If it's not something blatantly obvious that a monkey could figure out ( which it sometimes is ) then the chances are good that I have managed to drag in an old version of a library, mistakenly rolled back a file that didn't need to be rolled back or have something cached somewhere that brought it into the build without checking it. Going through my recent Source Control activity tends to reveal things I have done, cleaning the build often removes errant cached versions.
Sometimes it really is nothing to do with me - someone updated a dependency without mentioning it, WindowsUpdate installed something that changed the environment so that my code didn't work; there are a lot of background possibilities, but usually it's a case of manning up and accepting that, like most people, I am basically an idiot.
Use version control. Do a diff, or use your VCS' blame functionality.:
diff: Every VCS. Shows you the differences of, uhm, different versionsblame: for example git. Shows you on a line by line basis who has changed whatIf there is no version control, apart from it being your own or your boss's fault, you can look at the change dates of files and possibly look into your OS' logging facilities.
Apart from that: Recompile everything, make sure to also recompile auxiliary libraries.
Of course: If you've found the source of error, stay calm, ask for why a change was made, explain your problem, and propose a solution that makes you both happy. Don't shout at her/him, that would be poison to your productivity.
If there are no changes at all, it's time to see what has changed on the system. E.g., recently Mac OS computers have updated to a new version of Apache which has driven some configurations invalid.
git blame... had no idea it existed, but it's FCKING AWESOME
Commented
Jul 3, 2013 at 15:06
Well, here's a real life example of code that "worked yesterday" and not today... It is from earlier this month.
The application in question pulls information from a database by date, and the default behavior is to get data for the current day. This worked fine on August 8th, but failed on the 9th. It was not tested earlier than this. It would have also worked on 9th of September, and the 10th of October...
Another clue is that we are in the UK, the database in question was in the USA...
So, my answer to your question about what to check first is to double check how you format your dates, because if you mix up the day and the month fields it will work perfectly, but only on 1 day per month:-)
Fix the bug (however you normally do). Then if you find who caused it send them a polite email letting them know what went wrong.
Every coder makes mistakes and if you start blaming then it will seriously backfire the next time you do the same thing. (possibly even this bug was yours)
Its only if you suspect them of being reguarly careless should you make a big deal out of a couple of bugs.
...you run regression tests and focus on those that fail.
Actually is what you forgot to do yesterday before leaving, it happens.
You don't have any? Ok.. what where you saying? Blaming? Well... that might work, then
The first thing to do when something stops working is to ask yourself - What's different? What has changed?
When something worked last night but fails this morning, the one thing that obviously has changed is - the date and time :)
I'd try and think whether there's any part of the logic I'm working on that depends on dates and might be affected by the passing of time. It's surprising how many times that's the cause of such problems.
If that fails, you should definitely follow up on the other great advice supplied here.
A kinda short answer (to write) but kinda long to get the gist of it: Why Programs Fail: A Guide to Systematic Debugging by Andreas Zeller (which might look a bit too academic but it's not)
You look in your mailbox after the mail sent by the Continuous Integration engine when the unit test(s) failed (or the log page if you didn't watch that specific problem), and see who did the check-in just before that build.
Then go talk to him or her.
There are only two possible reasons why your code fails today, but worked yesterday.
Look at the data
There's something in the data you didn't test and or account for. Either data isn't validated properly or an error in logic wasn't revealed until a logical condition you didn't anticipate occurs. This means the bug was there yesterday, but it was hiding from you under valid data.
I once had some order entry code running fine for weeks. I went home one day, and it died. Investigation the next day revealed that I had a bug hidden in a chain of function calls. In a weakly typed language, I declared an integer when I should have used a long int. The language did the conversion between the two automatically until it couldn't because the number exceeded what would fit into an integer. The system failed on order number 32768.
Look at What Changed
Look at what changed since it worked. Did the IT section push out a OS update? Did another coder modify code that your program uses? Did the user's permission change? Often, if you find what changed, you'll find the bug.
works especially well for difficult JavaScript errors. Basically comment half the code, see if you get the error, if you do it's in that half of the code. Half it again and carry on.
If your code is well encapsulated this is a fantastic, time saving, stress busting tool.
Once you've found the guilty code, it's often worthwhile isolating the error on it's own test page.
And of course, what can be done to avoid being in such a situation?
Addressing this question, you might want to look into Continuous Integration (CI). Simply put: CI is a process where developers frequently (as many as several times a day) integrate and test all code. The idea is that changes to one module that break another module are quickly found.
In practice, most teams that employ CI use a CI Server (see: Wikipedia's list). The CI Server is usually setup to monitor the SCM repository and start a build when it sees changes. When the build is complete, it will then run a series of automated tests and post the results via e-mail and/or webpage of the build and tests, along with what changes caused the build. Hopefully, when something does break the build or tests, you have only a very small change set to look at, so it gets solved quicker.
There are other questions here about which CI Server to use, so I'll let you find them in interested. Personally, I am a big fan of Jenkins.
[What should I do about things being broken.]
As others have already said, find out what broke and try to fix it. Spending time trying to place blame is time spent not solving the problem.
My natural reaction is always to blame others but over time I have come to realise that it is usually me who is at fault. In addition to all the excellent comments above, it is important that you record for yourself what the final reason was. It doesn't matter whether you use a Wiki which is shared with other team members, a private Twiki, Evernote, a log book or a good memory. The important thing, at the moment you find the answer (and want to get back to work!) is to record the reason.
Presumably if it's no longer working, you have identified the symptoms of it not working, ie, it hangs, or throws back a particular error dialog to the user.
If the sole description of the problem is "it's not working", the first thing you need to do is gather more information on the symptoms of the problem.
Then you start looking for possible causes, either via logs or attempted recreation of the problem or a combination of both - depends on how your system is set up I guess.
Then you start ruling them out.
That's what usually happens when I take holidays :-)
More seriously, I'd first tell them:
I will look into it to see what's wrong and what could be the root
I will touch base in 30-60 minutes once I had a chance to see what's happening
After that time, I can hazard an estimate of what might have happened and how long it will take the fix it if it's not already fixed and, if applicable, what data we may have lost (but I have good backups, so that never happens hopefully).
As for the blaming part:
if it's just a colleague typo, there is no need to mention it: shit happens and the fright from the bug most likely taught him a lesson and hopefully, he won't do it again.
if he intentionnally did something I told him not to (e.g. give the root password of the production server to the new guy and tell him to make a modification on it directly without supervision)(yes, that already happened...), then I have to mention it.
If your usual bug tracing methods don't work and everything is a total mess, it can be wonderful to have a backup that you can restore easily.
This is what I run locally, automatically every hour from 8am to 6pm:
rdiff-backup /path/to/mystuff /path/to/mybackup
Simple, eh ?
If you ever have to restore anything, use
rdiff-backup -r 24h /path/to/mybackup/specific/dir /tmp/restored
rdiff-backup only stores files that differs. You can use rdiff-backup on Linux, mac and win.
Of course, this shouldn't be your only backup. But it's extremely easy and cheap way to have a local backup.
Now, I wouldn't recommend this as a normal bug fixing method, but if everything else fails, it's a fallback.
The bug may have already existed, but been hidden by external factors, or deep system issues.
This happened to me. A bug developed between 2 builds of our project. Literally, the only change we had made was to update to a more recent build of the one of the underlying libraries.
Naturally we blamed them. But the only change they had made was to refactor some headers for a faster compile. I agreed that that should not have broken the system.
After much debugging it turned out that the problem was a rogue pointer bug that had been latent in my code for years. Somehow it was never triggered until their refactoring had changed the arrangement of the executable.
it was working yesterday as it was being used correctly.
you find that other people use things in a way that there not suppose to which is a good way of breaking stuff.
its always good to update code early on in the day as this leaves you with a good testing environment.
Backup!
I find setting breakpoints to pause and check my data very helpful, to pinpoint exactly where and how it's going bad.