Biggest mistake you've ever made [closed]

Question

Similar to the question I read on Server Fault, what is the biggest mistake you've ever made in an IT related position. Some examples from friends:

I needed to do some work on a production site so I decided to copy over the live database to the beta site. Pretty standard, but when I went to the beta site it was still pulling out-of-date info. OOPS! I had copied the beta database over to the live site! Thank god for backups.

And for me, I created a form for an event that was to be held during a specific time range. Participants would fill out the form for a chance to win, and we would send the event organizers a CSV from the database. I went into the database, and found ONLY 1 ENTRY, MINE. Upon investigating, it appears as though I forgot an auto increment key, and because of the server setup there was no way to recover the lost data.

I am aware this question is similar to ones on Stack Overflow but the ones I found seemed to receive generic answers instead of actual stories :)

What is the biggest coding error/mistake ever…

Answer 1

At an internship, I was just learning SQL, performed an update without a where, affecting 17k+ rows instead of 10. Good news was that it was only the dev database. Bad news was that it couldn't be rolled back without the testers losing a lot of work, and a coworker was stuck fixing it for the rest of the day. I felt pretty bad.

Answer 2

We built an admin section on to a web application that shared messages between remote offices and the organization as a whole. Occasionally remote offices need to be shut down, or moved, renamed, etc. So we provided the standard CRUD operations for the list of offices. Sounds good so far, right?

When the application was going for acceptance testing in our client's staging area, our client started working with the admin section. They decided to remove one of the larger remote offices like "Paris" and the application started taking a looong time to respond. The client then attempted to readd the major office, and suddenly all the messages were no longer to be found.

The problem? We had cascading deletes turned on and thousands of important messages were suddenly gone forever! Well, we did have a backup, but had this gone into production it would have been a real embarrassment for both us and the client.

Back to the drawing board, and we basically had to build some protections to move the messages over to a new remote office or hold them in a temporary bin until they could figure out where to put them.

Answer 3

We were working around a problem in flagging exceptions in an ETL process. The solution I proposed was an ugly hack where we would delete a row from table A when that data was written to table B, where the exceptions would be logged. The code I wrote deleted everything from table A when a single row was added to table B. That resulted all data before the first exception being deleted on load.

It went into production. It ran for two weeks before someone caught the bug.

Doh!

Answer 4

Was attempting to do a quick'n'dirty backup of a Solaris box and inadvertently replaced every file in the file system with a compressed version of itself. Solaris really hates you doing that in places like /etc!

This was actually part of a nightmare week for me - I blogged it a while ago - http://blog.superpat.com/2006/02/09/best-week-worst-week/

Answer 5

This wasn't an error by me but by a coworker. He was programming on a remote server on a piece of code that wasn't under revision yet. He wrote around 600-800 lines of code in that day... He was programming in Vim or Emacs.

When he finished his work, he accidentally erased the content of the file and saved and exited the editor.

We got lucky and our sysadmin made a dump of the hardrive to recover saved data before that space get reused by an other program. Luckily he did save that file periodically and that means data was on the hardrive but not associated to the file.

Answer 6

Several years ago in the army, I was in Company C (leaving out my full unit name so as not to embarrass them). Company B with about 30 soldiers had just spent the past month entering data on soldiers into a new system manually. I then accidentally deleted the DB and it was not recoverable and they had failed to make a backup the entire month (so partially their fault). But these 30 soldiers had to spend the next month repeating doing the same thing they had just done the previous month. They were not happy soldiers. I had to be careful where I went over the next few weeks as these other soldiers were ready to declare war on one of their own!

Answer 7

While working on designing a new website for a collegiate entity, I accidentally deleted the entire code base. There was no version control. Luckily, they took nightly backups so one quick email to the IT guys and 5 minutes later I was back in business, recovering maybe 25 minutes of lost work. Thank god it happened first thing in the morning and not right before I left...