Methods of debugging code (Nightmare situation)

Question

I am frequently tasked with debugging an application at my job. It is a BI Application that we deploy to businesses, which includes a test environment, and a production environment. I am wondering if there are any apps/tools/methods that people can suggest, based on these constraints:

1. Debugger cannot be used on client site or locally, because the software depends on custom third party applications that we don't have test environments for. (EDIT: to be fair, it is possible to debug locally in some cases. If we use nothing but the core code. Much of the problematic code resides in a dll that encapsulates third party specific communication: sockets, process pipes, soap calls, custom logic that changes the behavior of core code. Typically during an implementation or enhancement for a client, we would be writing new code to this area.)

2. There is virtually no logging done in our apps. There are no unit tests.

3. Version control only has 1 version of the full solution (using source safe 2005). So it's not possible to get a previous version of the entire solution, only individual files. (Unless someone knows ways around this).

4. Cannot reproduce locally, often times cannot reproduce on test environment (High chance that test and production are not the same version).

5. There is a high chance that the version the client is using is different from the one on source safe. This is because individual files are updated, which have embedded custom logic for that specific client. Often what happens is an update is made to a binary, which requires changes to several other binaries, but when a commit is done, no one has any record or knowledge of this. A somewhat common error I see is the 'Function/Method not found' or 'Method call has too many/too few parameters specified' on a clients environment.

6. This is a .net VB solution

7. Cannot install any software on client sites, but can locally

8. Our application is extremely customizable, but unfortunately the customization logic is spread out across all the classes and files, from the front end all the way to the data layer, including custom changes made to the database on a per client basis.

9. There are virtually no comments in the code. There is no documentation about the architecture. No documentation about the api. The only thing we have are hundreds upon hundreds of email chains that somewhat explain what's going on. The only people that know the code are the ones that originally wrote it, but they're no longer developers per say so they don't get involved that much.

And before you say it... yes I know; I want to shoot myself as well. It doesn't help that there's spaghetti code, hundreds of compiler warnings, and broken polymorphism that REALLY should be fixed, but I don't have a say in it.

Most common sort of errors I run into are null reference errors, invalid casts, and missing functions/function signature mismatches. Sometimes I am lucky and event viewer will log the class, method, and exception message. It's not the most helpful, but it's still something. The worst are the errors that have no trace, no repro steps besides a screenshot, and are generic error messages like the ones mentioned above. Sometimes it's not possible to find out why they occurred, only to pray that the environment is not properly configured, and that it will go away later.

I know this comes off as a bit of a rant, and to some extent it is. But I'm desperate for options. Are there other methods / tools I can use?

Answer 1

Robert Harvey's advice is likely best, but since career advice is off topic, I'll give what answer can be given:

You are at the bottom of a very steep mountain covered in brambles and mud and irritable mountain goats. There's no easy way up. If you want to get to the top, you've got to force your way up one tremendously painful step at a time.

It seems you know exactly how things should work. If no one else will help you then (again, ignoring career advice) your only choice is to start fixing these things yourself.

First, for all that is holy, get that stuff in a real version control system. Pretty much anything at all but Source Safe, which is well-known as a stinking pile of garbage. git is free and can be set up fairly easily. You can't fix issues of the past lack of version control, but at least stop the problem from continuing on into the future.

Next, look into logging. Find, or worst case, write a logging system, and start using it. Use one that can be used on the client sites as well, so when things go sideways you have at least something.

And start writing tests, at least for new changes you make.

There is no sugar coating it: there is no answer here that doesn't involve lots of work, or treating this as a question of career.

Answer 2

1) Debugger cannot be used on client site ...

That's perfectly normal.

... or locally

Now that's a problem.

2) There is virtually no logging done in our apps.

Logging is the Production Debugging.

There are no unit tests.

Oh dear. All to common, though.

3) Version control only has 1 version of the full solution

Then you're not using Version Control.

4) Cannot reproduce locally, often times cannot reproduce on test environment (High chance that test and production are not the same version).

So only the deployed, client environment shows the errors.

In that case, you need disgnostic logging embedded in the application code that (a) traps and [fully] records [fatal] errors and (b) can be "dialed up" on demand to produce additional, continuous, diagnostics that are useful in tracking down the problem(s).

5) There is a high chance that the version the client is using is different from the one on source safe.

Again, you're just not using version control to your advantage.

8) Our application is extremely customizable

That's fairly typical.

The site-specific differences should be managed through Version Control "Branching".

9) There are virtually no comments in the code.

Again, that's all too common, because Developers write "self-documenting" code, don't they?
Or, at least, code that they understand on the day that they write it.

There is no documentation about the architecture.

Oh dear.

No documentation about the api.

Oh dear, oh dear.

And before you say it... yes I know; I want to shoot myself as well.

No; you want to shoot the people who wrote all this stuff, created an undocumented, unmaintainable mess and then moved on to pastures new, leaving the unconscionable mess behind them.

Most common sort of errors I run into are null reference errors, invalid casts, and missing functions/function signature mismatches.

Null references and invalid casts are run-time errors; to some extent, you should be expecting them and the fact they you're getting them frequently suggests a lack of defensive programming (or a surplus of optimism) by the original authors.

Function signature mismatches should cause a broken build; those should cause "broken builds" and should never make it out of the door!

Answer 3

Start with logging. This will have the greatest impact. Implement a logging framework into the code base, like Log4Net or similar. Start logging what the code does.

Debugging should be possible locally. If not, work on getting the symbol files (PDBs) so you can debug into 3rd party dlls to gain a complete picture of the issues that are occurring. Tools like WINDBG can point which DLLs are problematic if the system is crashing. One can configure any server to take a memory dump when there is a crash. It's basically a snapshot of what was going on when the problem occurred. One can examine the dumps locally to find clues to what was happening. If debugging is not possible, work on making it possible. Document the steps necessary to debug. Sometime on complex systems, there is quite a lot of setup necessary to fully debug.

Bug tracking...If you not using one, start using one. This goes hand and hand with a proper version control system. Basically, start tracking defects and revisions of your code. Start to build a history of the system.

Perform Static Code Analysis. Invest in a tool like ReSharper. It will quickly point out all possible null reference exceptions and other bad coding practices. It can help get the code in better shape with just a few clicks and automate tedious items like code formatting, variable naming, etc. Measure your code, find out where the hot spots for refactoring via code metrics.

Refactor and Unit tests. I'm going to assume that probably most of the code written is not very testable, so I wouldn't bother trying to add tests for it. Any new code, create a tests project and start writing tests, both unit and integration. If the unit tests fail, fail the build. So, as you refactor, there should be tests. One thing with tests is that one can write a test to call any method and debug into that method without loading up the entire application or code base. This is useful to help troubleshoot issues.

Document any tribal knowledge as needed. The code should be self documenting so comments so be sparse, but many system have "unusual" ways of doing things, point those out in a coding WIKI or other type of informal repository. Also, consider coming up with coding standards and practices. Enforce those standards via a toolset like Resharper. Since most of the code is probably not following any standard and guidelines, implement the standards on new code that is written.

Since your new, I would treat this like a tour of duty. 6 months to 2 years, and then make the choice to stay or move on. Take satisfaction from making things slightly better than the day before.

Answer 4

First, all the above... ditto.

Some heuristics:

• Use source control on your development computer. It's the best thing I've done. It is not a substitute for the project's version control, which we have. It's a tool that gives amazing freedom to fearlessly experiment, hack, work problems simultaneously yet independently, etc. I am better at using version control because I have the freedom to be bold, screw up, and learn.
  • What's the difference between Git and Mecurial?
  • Git is Mcgyver, Mecurial is James Bond
• To the extent you add comments, prioritize the public interface elements, to take advantage of intellisense. Do this as you decipher during your debug adventures.
• Be persistent with minor refactorings. Sooner or later, for a given hunk of code, you will get to a critical mass that enables big refactorings like DRYing up redundant code across classes.
• Do not mix code reformatting and actual changes in the same version control commits.
• SOLID principles
  • Never ignore single responsibility. Well done, this is the path to the promise of object oriented; IMHO.
  • Always ignore Open/Closed.
  • We're not talking new code here.
  • Making interfaces without purposeful design hinders maintenance.
• Refactoring
  • There is a methodology to it. Buy the book.
  • Refactoring: Improving the Design of Existing Code
• Some code files need complete reformatting, variable renaming, etc. before even attempting debugging. Don't be shy about using visual studio's refactor menu without unit tests.
• The only documentation that can't be misplaced is in the code files.
• When you do get version control give forethought to VC plan. And document it! And come up with a naming convention for branches, tags that will highlight major software versions and milestones.
• Use good equipment
  • Get a monitor that pivots. Vertical is the way to read code quickly.
  • External drive for backups, keyboard with the right kind of keys, multi-function mouse. At least 1 really good USB drive. I even have my own Herman Miller chair.
  • Get very good comparison software. I'm happy with Beyond Compare.
• Practice Rubber Ducky Debugging
• Sometimes the worst thing that can happen is that they don't fire you.

Edit

Brownfield Application Development in .NET

Try this book. An initial cover to cover read through is probably best. This book will help your thinking about the big picture, possibilities, and developing both strategic and tactical plans of attack.

Sticking it out

Stay, say, 1.5 years if you can; long enough to know if you are making experiential progress. You will know if you are getting 2 years of experience or 6 months of experience 4 times.

Being "junior with 1/2 years experience" I'm concerned that a potential employer will view it as bailing out early because you couldn't hack it. It's one thing to say you learned z, y, x, took some names and kicked some ass - but was not allowed to contribute to your capabilities; and another simply risking dissing the job, the code, management, etc. by way of explanation.

I may be off base on this but my "best of times and worst of times" was my 1st job, which happened to be literally unmaintainable code. I had a great supervisor (all the rest were from the management in-breeding program) who gave me the space to re-write some key programs. That experience was revelation.

end Edit

Answer 5

I would say (5) is the one you need to fix first. If you don't know which code is running in production, you have no safe way of reproducing and fixing problems. This makes any other change you introduce dangerous, since it might cause problems you cannot foresee and cannot reproduce.

You may need to do some detective work and perhaps reverse engineering to figure out which version(s) of code and which libraries is deployed. (And you need a have a consistent build and deploy system to bring all deployed code in line with source control going forward.)

You might have to build multiple test environments to replicate the various deployments. (Of course the simplest fix is to create a new clean build and deploy it consistently everywhere, but it sounds like this is not possible?)

Only when you know the exact versions deployed and have corresponding test environments, should you start trying to fix/improve the code.

Your next priority should be to consolidate to a single code base which can be deployed everywhere. It sounds like you have various versions of code deployed due to customization? You should consolidate to a single version and then use configuration switches for the custom logic.

After this, you can start to carefully improve to code base to allow easier debugging. Adding logging is probably the least risky improvements.

You will want to add automated tests, but unittests are often hard to add to a project which is not initially designed for testing. Instead I recommend starting out with automated end-to-end integration tests. These are more tricky to set up, but does not require you to re-architect the solution, so are less risky.

Answer 6

Ignoring the issues you have in your team, it seem that the first one to address is to debug the code matching what is in production. Otherwise you might be chasing a bug that was already fixed in the code you have in your "Source control". As this is .NET you could easily "decompile" the production binaries to compare the code with what you have. It's not an easy task but if you succeed this is a strong argument for a better source control tool that can tag released versions.