How do you dive into large code bases?

Question

What tools and techniques do you use for exploring and learning an unknown code base?

I am thinking of tools like grep, ctags, unit-tests, functional test, class-diagram generators, call graphs, code metrics like sloccount, and so on. I'd be interested in your experiences, the helpers you used or wrote yourself and the size of the code base with which you worked.

I realize that becoming acquainted with a code base is a process that happens over time, and familiarity can mean anything from "I'm able to summarize the code" to "I can refactor and shrink it to 30% of the size". But how to even begin?

Answer 1

what I've always done is the following:

Open multiple copies of my editor (Visual Studio/Eclipse/Whatever) and then debug and do line breaks step through the code. Find out the flow of the code, stack trace through to see where the key points are and go from there.

I can look at method after method - but it's nice if I can click on something and then see where in the code it's executed and follow along. Let's me get a feel for how the developer wanted things to work.

Answer 2

How do you eat an elephant?

One bite at a time :)

Seriously, I try to talk to the authors of the code first.

Answer 3

Do I have to hack till I get the job done

To a large extent, yes (sorry).

Approaches you might consider:

1. Try to find out what the code is supposed to do, in business terms.
2. Read all the documentation that exists, no matter how bad it is.
3. Talk to anyone who might know something about the code.
4. Step through the code in the debugger.
5. Introduce small changes and see what breaks.
6. Make small changes to the code to make it clearer.

Some of the things I do to clarify code are:

1. Run a code prettifier to format the code nicely.
2. Add comments to explain what I think it might do
3. Change variable names to make them clearer (using a refactoring tool)
4. Using a tool that highlights all the uses of a particular symbol
5. Reducing clutter in the code - commented out code, meaningless comments, pointless variable initializations and so forth.
6. Change the code to use current code conventions (again using refactoring tools)
7. Start to extract functionality into meaningful routines
8. Start to add tests where possible (not often possible)
9. Get rid of magic numbers
10. Reducing duplication where possible

... and whatever other simple improvements you can make.

Gradually, the meaning behind it all should become clearer.

As for the place to start? Start with what you do know. I suggest inputs and outputs. You can often get a handle on what these are supposed to be and what they are used for. Follow data through the application and see where it goes and how it is changed.

One of the problems I have with all this is motivation - it can be a real slog. It helps me to think of the whole business as a puzzle, and to celebrate the progress that I'm making, no matter how small.

Answer 4

Your situation is actually common. Anyone having to walk into a new job where there is existing code to work with is going to deal with some element of it. If the system is a really nasty legacy system, then it is very much like what you've described. Of course, there is never any current documentation.

First, many have recommended Working Effectively with Legacy Code by Michael Feathers. This is indeed a good book, with useful chapters like "I can't get this class into a test harness" or "My application has no structure" though sometimes Feathers can only offer more sympathy than solution. In particular, the book and its examples are largely geared to curly braces languages. If you are working with gnarled SQL procedures it may not be quite as useful. I think the chapter, "I don't understand this code well enough to change it," speaks to your issue. Feathers mentions here the obvious things like taking notes and marking up listings, but also makes a good point that you can delete unused code if you have source control. Lots of people leave commented sections of code in place, but that often isn't of help.

Next, I think your suggested approach is certainly a good step. You have to understand first at a high level what the purpose of the code is.

Definitely work with a mentor or someone on the team if you have to get questions answered.

Also, take the opportunity to support the code if defects are revealed (though sometimes you don't have to volunteer for this...the defect will find you!). Users can explain what they use the software for and how the defect is affecting them. That can often be a very useful bit of knowledge when trying to understand the meaning of the software. In addition, going into the code with a purposeful target to attack can sometimes help focus you when facing "the beast."

Answer 5

I like to do the following when I have a really large source file:

• Copy the whole mess onto the clipboard
• Paste into Word/textmate whatever
• Reduce font size to the minimum.
• Scroll down looking at the patterns in the code

You would be amazed at how oddly familiar the code looks when you get back to your normal editor.

Answer 6

It takes time

Don't feel too rushed while trying to understand a legacy codebase, especially if it's using technologies/languages/frameworks you're not familiar with. It's just an unavoidable learning curve that takes some time.

One approach is to go back and forth between the code and tutorials on the related technologies. You read/watch the tutorial, then go look at the code to see how your predecessors did it, noting any similarities and differences, taking notes, and asking questions to any existing developers.

"Why did you do this part this way"

"I noticed most people online are doing it this way, and you all did it another way. Why is this?"

"What made you all choose technology X over technology Y?"

The answers to these questions will help you understand the history of the project and the reasoning behind design and implementation decisions.

Eventually, you'll feel familiar enough with it that you can start adding/fixing things. If it all seems confusing or it seems like there's too much "magic" going on, you haven't spent enough time looking it over, digesting it, and diagramming it. Creating diagrams (sequence diagrams, process flow diagrams, etc.) is a great way for you to understand a complex process, plus they will help the "next guy".

Answer 7

cscope can do whatever ctags can do for C, plus, it can also list where all current function is called. Plus it is very fast. Scales easily to millions of LOC. Integrates neatly to emacs and vim.

C and C++ Code Counter - cccc can generate code metrics in html format. I have used wc also for getting LOC.

doxygen can generate syntax highlighted and cross referenced code in html. Useful in browsing large code base.

Answer 8

The way I recommend with Drupal and it's not really Drupal specific: start with the issue tracker. There will be old, unclosed bug reports for sure. Can you reproduce them? If yes, update the ticket confirming it. If no, close it down. You will find this way a ton of ways to use the software and you can begin peeking into the codebase where it crashes. Or you can begin to step through the code and see how it arrives to where it crashes. This way you will not just begin to understand the codebase but also accumulate a ton of karma and your questions will be warmly welcomed by the community.

Answer 9

One important thing to do is to use tooling to generate dependency graphs to top-down explore code architecture. First visualize graph between .NET assemblies or jars, this will give you an idea on how features and layers are organized, then dig into namespaces dependencies (inside one or a few relatives .NET assemblies or jars) to have a finer grained idea of code structure and finally you can look at classes dependencies to understand how a set of classes collaborates to implement a feature. There are several tools to generate dependency graph, like NDepend for .NET for example, that generated the graph below.

Answer 10

I once had a pretty fantastic software engineer tell me that the most expensive form of code analysis and maintenance was to walk through the code, line by line; of course, we're programmers, and that pretty much comes with the job. The happy medium, I think, is to (in this order): 1. Get a notebook to create notes for how you understand the code to work, and add to it as time goes on 2. Refer to documentation about the code 3. Talk to authors or others who have supported the code base. Ask them for a "brain dump" 4. If you're to the point where you understand some of the detail-level class relationships, do some step-through debugging of the code to do some synthesis between how you thought the code works and how the code actually works.

Answer 11

First understand what it's meant to be doing - without that it's likely to be gibberish. Speak to the users, read the manual, whatever.

Then hit run and start walking the code for what seem to be the key functions.

Answer 12

Divide and conquer. I look at each functionality and the associated code, step through them and go on to the next, slowly building a picture of the whole.

If the project had unit tests, I like going through them too, they're always very revealing and enlightening.

Answer 13

1. Run all the tests, if you have any, and see which code is covered and which isn't.
2. If code that you need to change isn't covered, try to write tests to cover it.
3. Change the code. Don't break the tests.

See Michael Feathers' Working Effectively with Legacy Code

Answer 14

Here's my short list:

1. If possible, have someone give a high-level view of the code. What patterns were considered, what kinds of conventions could I expect to see, etc. This may have a few rounds to it as in the beginning I'd get one story that as I get more familiar with the code, I may have new questions to ask as I work through the onion of the pre-existing project.

2. Run the code and see what does the system(s) look like. Granted it may have more than a few bugs, but this can be useful for getting an idea of what does it do. This isn't about changing the code, but rather just seeing how does this run. How do various pieces fit together to be a system overall?

3. Look for tests and other indicators of basic documentation that may assist in building an internal mental model of the code. This is where I'd probably suggest at least a few days unless there is extremely little documentation and tests of course.

4. How well do I know the languages and frameworks used in this project? The importance here is the difference between looking at some things and going, "Yes, seen that a dozen times before and know it fairly well," and "What in the world is being attempted here? Who thought this was a good idea?" kind of questions that while I wouldn't say them out loud, I would be thinking them especially if I'm looking at legacy code that may be quite fragile and the people that did write it are either unavailable or just don't remember why things were done the way they were. For new areas, it may be worthwhile to spend some extra time getting to know what is the structure and what patterns can I find in this code.

Last but not least: Know the expectations of those running the project in terms of what you are supposed to do at each point in time, given the following few ideas of what may be expected:

• Are you putting in new features?
• Are you fixing bugs?
• Are you refactoring code? Are the standards new to you or are they very familiar?
• Are you supposed to be just familiarizing yourself with the code base?

Answer 15

I always try and start with the entry point into the program, since all programs have one (Eg main method, main class, init, etc). This will then point me to what gets started and sometimes how things are linked.

After that I drill down. The database and DAO are configured somewhere, so I get a sense of how things are stored. Perhaps some kind of global instance class is started as well, and there I can figure out whats being stored. And with good refractoring tools, I can find out who calls what.

I then try and find where the interface is configured and handled, since this is the next entry point of information. Refractoring, searching, and debugging tools aid in my search. I can then figure out where the handling of information starts and ends, working my way through all the class files.

I then try and write the flow down on some paper, just to initially wrap my head around things. The submit button passes to the generic verification which then gets passed to the DAO or database and then gets stored in the database. This is a gross oversimplification of most apps, but its the general idea. Pen and paper are extremely helpful here, since you can jot down everything quickly and not have to worry about formatting in a program that was supposed to help you.

Answer 16

I would say to start with documentation, etc., but in my experience, the depth of documentation and local knowledge is often inversely proportional to the age, size and complexity of a system.

That being said, I usually try to identify a couple of functional threads. By functional I mean stuff like logging in, pulling down a list of customers, etc. If the patterns are consistent at all, one thread should give you a nice, not necessarily complete, cross-section of the system. The best way to determine if the patterns are consistent is to analyze a handful of threads.

I think this goes without saying but, in my opinion, it's better to understand the system from a functional perspective rather than from a technical perspective. I generally don't worry too much about the tools that are being used (ORMs, logging libraries, etc.) and focus more on the patterns (MVP, etc.) that are being used. In my experience, tools are generally more fluid that patterns.

Answer 17

I done so much...

Here is my current approach for situations when there is "something working", and you need to make it "work in some other way".

1. Get goals, that system should solve (if they not written) - write it. Ask manager, other employees, even former if they available. Ask customer or search any piece of documentation.
2. Get specificatio. If it not exist - write it. It is not worth to ask someone for it, as if it is not exist, then you are in situation when other don't care much. So only way to write own (later it will be much easier refer to it).
3. Get design. Not exist - write it. Try to refer to any documents and source code as much as possible.
4. Write detailed design to part that you need to change.
5. Define how you test it. So you could be sure that old and new code work in same way.
6. make system be able to be build in one step. And test with old code. Put it to SVC if it is not already.
7. Implement changes. Not earlier.
8. verify after a month or so, that nothing is broken.

One more optional todo that may require between each step: f off manager (project owner) who tell you that "these changes should be done already yesterday". After few projects, he may even start helping to get specs and docs in advance.

But usually (especially for scripts) it just not possible in business scope (cost will be too high, while value will be to low). One option is not to do any changes, until critical mass reached, and system goes off from production (e.g. new system will come) or management decided that all these is worth to do.

PS: I remember one code that were used for 5 clients with different settings. And each change (new feature) were required thinking of "what parts are used", and "what configuration clients have" so not to brake anything, and not to copypaste code. Putting their settings to project cvs, and writing specs, reduces this thinking time almost to 0.

Answer 18

Print out the source code and start reading through it. If it's especially large, only print out select portions of it to better understand it and make as many notes/comments as you need.

Trace through the program starting from the beginning of its execution. If you are assigned to a particular portion of the code base, trace the execution within that portion and figure out what data structures are used.

If you're using an object-oriented language, try to make a general class diagram. This will give you a good high-level overview.

Unfortunately, in the end, you will have to read through as much code as possible. If you're lucky, the previous programmers have written as much documentation as possible to help you understand what's going on.

Answer 19

The first thing you need to do when learning a new code base is learn about what it is supposed to do, how its used, and how to use it. Then start looking at architectural documentation to learn how the code is laid out, also look at how the database at this point. At the same time you are learning the architecture its a good time to review any process flows or use case documents. then start diving in and reading code after you have an understanding of the big picture, but only code related to any work you may be doing on this code, don't just try to read all the code. Its more important to know where the code is to do X than exactly how X is done, the code is always there to tell you how if you can find it.

I find that just trying to jump in and read code without a goal beyond learning the code is generally unproductive, trying to do small changes yourself or reviewing the code of someone else's changes is a far more productive use of your time.

Answer 20

If a code base is large, then focus your attention to the parts that are currently working on. Otherwise you will feel overwhelmed and possibly your head may explode. I think some high level overview is helpful (if it is available), but chances are you will be spending lots of time in the debugger for following program flow. It is a good idea to get an overview of the application and seeing it used, so you can understand how/what/why the code is being used for.

I usually run some sort of code complexity tool on the code to tell me where the problem areas are. Areas that score high are probably very hard to update. For example, I ran into a function that scored 450 on the cyclomatic scale. Sure enough, hundreds of IFs. Very hard to maintain or change that. So be prepared for the worst.

Also, don't be afraid to ask questions to existing developers, especially if they worked on the system. Keep your internal thoughts to yourself and focus on solving the problems. Avoid comments that may make the other developers becoming upset. After all, it may be thier baby and nobody likes being told thier baby is ugly.

Take small steps, even the smallest code change can have big impact.

I find it is helpful to come up with program code flows so if I am making changes, I can do dependency searches to see which methods/functions call what. Suppose I am changing method C.

If only 1 method/function calls C, then it is a pretty safe change. If 100s of methods/functions call C, then it would be of greater impact.

Hopefully your code base is well architected, written, and maintained. If so, it will take some time to understand it but eventually the tide will be turned.

If it is a big ball of mud, you may never understand (or want to understand) its inner workings.

Answer 21

Some things I do...

1) Use a source analysis tool like Source Monitor to determine the various module sizes, complexity metrics etc.. to get a feel for the project and help identify the areas that are non-trivial.

2) Drill through the code top to bottom in Eclipse (good to have an editor that can browse references, etc.) until I get to know what's going on and where in the code base.

3) Occasionally, I draw diagrams in Visio to get a better picture of the architecture. This can be helpful for others on the project as well.

Answer 22

This happens a lot. Until I started to work on an open source platform I don't think I ever started a job which didn't start out with an admission that the code had some 'quirks'.

You can get a long way with a step debugger and a lot of tenacity. Unfortunately often it takes time and experience to learn a particular big ball of mud and even then after years there can still be some subsystem which crops up which nobody has any knowledge of.

Answer 23

I'd encourage you to write unit tests before you change anything in the ball of mud. And only change enough of the code at the time to make the tests pass. As you refactor, add unit tests before hand so you know that the business functionality hasn't been busted by refactoring.

Is pair programming an option? Having another person to bounce ideas off is a great idea to deal with that amount of nasty.

Answer 24

Here is a procedure we use for eliminating duplicates.

• select a standard comment prefix for duplicates (we use [dupe] right after the comment marker;
• write specs with your teams about names to use for the duplicate procedure;
• first round: everybody takes some files, and adds [dupe][procedure_arbitrary_name] before the duplicated procedure;
• second round: everybody takes a procedure, or a subset of procedure, and assigns a value indicating the order of likeability of the different same-purpose implementations (the string will then be: [dupe][procedure_arbitrary_name][n]);
• third round: the responsible for each procedure rewrites it in the relevant class;
• fourth round: grep happy!

Answer 25

I think one of the most important things is to take a simple feature, choose the simplest you can think of, and implement it. If there is a maintained wish list use that or else talk to someone familiar with the code base and get them to suggest a feature. Usually I would expect this to be a change with 5~20 LOC. The important point is not that you are adding a very fancy feature but that you are working (or rather grappling :) ) with the code base and going through the whole workflow. You would have to

1. Read code to understand the component that you are modifying
2. Change code and understand how that impacts the surrounding system.
3. Test the change and thus identify how the components interact with each other
4. Write the test case and hopefully break one or two test cases so you get to fix them and understand the invariants of the system.
5. Build the thing or see the CI build it and then ship it

The list goes on but point is that a mini project like this takes you through all of the items on your check list for getting acquainted with a system and also results in a productive change being done.

Answer 26

A little thing I wanted to add:

One tool that I've started using recently for this kind of problem that has helped immensely is mind mapping. Instead of trying to cram all the details of how something is implemented in my head, I'll build a mindmap describing how the system I am going through works. It really helps me more deeply comprehend what is going on and what I still need to figure out. It also helps me keep track of what I need to change at a very precise scale.

I recommend using free plane amongst the plethora of mind mapping choices.

Answer 27

There won't be any documentation or there will be scant documentation, or it will be out of date. Find all the documentation that does exist. If it's in a team repository, don't make a copy. If not, put it there and ask your manager for permission to organize it, perhaps with some supervision.

Get everything up into the repository for the team and add a Glossary. All bases have jargon; document it in the glossary. Make sections for tools, products, customer-specific, etc.

Create / Update a software environment creation document. All tools, quirks, install choices, etc. go here.

Then upload a Getting Started With "ProductName" document or the like. Let it be just mind flow and self-organize over time. Then go through out of date docs and get them back to date. The other developers will appreciate it, you'll be contributing in a unique way whilst learning the code. Especially document all such things that stump you or are named wrong or are counter-intuitive.

Once your leaning curve is coming to an end, don't worry about updating the documentation. Let the next new guy do that. When he arrives, point him to your work. When he continually bugs you for answers, don't answer him. Rather, add the question to your documentation and then hand him the url. Fishing pole.

One side-effect is that you will have made a tool you yourself can reference months from now when you forget.

And although it's not documentation, a related issue is all the little quirky, manually intensive procedures your team mates do. Automate them with batches, sql scripts and the like, and share those as well. After all, procedural knowledge is arguably as big as declarative knowledge in terms of getting productive in a new environment. Whatever it is, don't do it; rather, script it, and run the script. Fishing pole strikes again.

Answer 28

I wrote a fairly lengthy post on this topic. Here is an excerpt

I thought about this problem for a quite while. I decided to write up my own personal solution as a general process. The steps I have documented are as follows:

1. Create Vocabulary Sheet
2. Learn the Application
3. Browse Available Documentation
4. Make Assumptions
5. Locate Third Party Libraries
6. Analyze Code

This process is written in the context of a large desktop application, but the general techniques are still applicable to web applications, and smaller modules.

taken from: A Process For Learning A New Codebase

Answer 29

There are few small tips I can share.

For an existing product, I start testing them intensively. If pick/get assigned a task, I will focusing on the particular feature more.

• The next step would be finding the code where I can break in and start exploring On the way I will find the dependent modules, libraries, frameworks etc.

• Next step would be creating simple class diagram with it's responsibilities (Like CRC cards)

• Start making minor changes or take up minor bugs to fix and commit. So we can learn the work flow of the project; not just the code. Often large products will have some sort of book keeping for the sake of authorization and audits (e.g. healthcare projects)

• Talk to the people who are already working on the project. Express your ideas, thoughts and in return get their experience and views on working with this project for long time. This is quite important because that also helps you get along with the team well.

Answer 30

It has been a long time since I have had to dive into a large code base myself. But in the recent couple of years I tried many times to get new developers into teams where we had an existing, rather large, code base.

And the method that we have used successfully, and I would say is the most effective way without question IMHO, is pair programming.

In the last 12 months, we have had 4 new members to the team, and every time, the new member would pair with another member well acquainted with the code base. In the beginning, the older team member would have the keyboard. After about 30 minutes we would pass the keyboard to the new member, who would work under the guidance of the older team member.

This process has proven to be quite successful.