I program a data-analysis framework for my company (20000LOCs by now). There are 2 programmers who help writing individual module for interfaces I have predefined. So far I haven't used much planning or tools. I basically think of new classes in my head and write them down in Python+Eclipse. As Python is good for proto-typing I can easily modify it if I overlook one aspect. However, sometimes I need to refactor some structure and I wonder if that hadn't happened if I used planning tools.

What do you think is the most important start to plan programming more professionally? The project won't become too big as it's only for data analysis, but managing which data is processed which way is also tedious. Can you suggest methods, reading or tools?

My class hierarchy is pretty flat so I don't see much use in class diagrams. However, something like call diagrams and diagrams which show how objects reference each other might help. I've learned OOP by myself, but I'm confident that once I consider all requirements I can design a nice class structure. What about TDD? For the moment I cannot imagine how to use it since most calculation are on network-like data with objects and many object references (links). Moreover data content hasn't been a problem yet.

Yet, I suspect that some more professional approach might be more efficient. Ideas?

  • 1
    how does your project score at The Joel Test? "A score of 12 is perfect, 11 is tolerable, but 10 or lower and you've got serious problems..." – gnat Feb 13 '12 at 12:37
  • @gnat The Joel Test works for Joel, but of itself isn't really definitive in terms of measuring how well a company or individual performs under other (non-Joel) conditions. That said, I'd agree with most of the options on the Joel Test, but could happily live without a few while adding a several different options of my own. Totally in keeping with my Agile leanings. ;-) – S.Robins Feb 13 '12 at 12:56
  • 2
    With decades of experience, I'm confident that once you consider all requirements you still not be able to design the nicest class structure. You will over-design some parts, and under-design others, and will still need to refactor as you code. People are just not smart enough to produce perfect code in a single pass. If we were that smart, we wouldn't even need computers, because we could just compute everything in our head. – kevin cline Feb 29 '12 at 1:11
  • possible duplicate of Should I plan ahead, or figure out programs as I'm writing them? – gnat May 20 '15 at 22:04

The fact that you are sometimes refactoring code is not a bad thing; being prepared to do so shows a commitment to writing the best code you can.

It sounds like you are looking for a more rigid way to define software up front. Certainly thinking through and planning is important, but moving too far in the direction of Big Design Up Front (BDUF) has its own problems. These include not knowing all the requirements in advance, and not being flexible enough to change the design if (when) the requirements change. Like many things that are hard it's a trade-off between two extremes of a pendulum. You might find it interesting to read about the Waterfall Method and it's problems.

The limited size of your project means a dynamic approach has more up than down sides. Refactoring your way out of any wrong turns is probably not going to be difficult.

I would encourage you to have another look at TDD. It is not that the code is proven correct (it isn't, although it will provide more evidence that it is). The main advantage is that it gives you the confidence to refactor more. The idea is the tests are green, you change the implementation and break the tests, then get them green again. If you have good tests it avoids those nasty situations where you try to change to much and get yourself into trouble and can't work out why things are broken (of course if you are using source code control rolling back is a command away).

As for it being difficult to test you can use mock objects. These allow you to isolate classes for testing which are strongly entwined (highly coupled). If you follow TDD you will find that you naturally write classes that stand alone. This makes testing easier, but happily it also makes the code better (low coupling, and high cohesion).

If you are interested in this I second the recommendation for Martin Fowlers book Refactoring: Improving the Design of Existing Code.

  • Thanks! Very interesting post. I wouldn't go a far as BDUF as when refactoring I noticed that I just couldn't forsee all usability dimensions even though I tried hard. And since IT students told me that you always write code 3 times, I'm more confident. I will follow your suggestion and look how to refactor more organized. As for TDD, I'll have one more look. However data-wise I haven't had major bugs yet. Yet, sometimes after refactoring, calls in other modules break which might be due to me not using all Eclipse refactoring features. – Gerenuk Feb 28 '12 at 10:25

Is how you have been doing things working? Have you been successful meeting all of your deadlines and satisfying all of your customers? Is this the way you have always worked and are you planning on working with other software developers with slightly different expectations of you?

I could give you lots of recommendations about how I personally would be doing things, but the question really is what you and your team feels really works for you. If you are asking these questions merely out of interest, then perhaps you aren't yet ready to change the way you do things. If on the other hand you have been feeling that you're tasks and workload have been getting the better of you, and you feel that they way you work isn't working, then perhaps you need to review your methodology and deal with these problems at a much higher level first.

Planning tools aren't going to help you to decide what needs refactoring. Those tools are merely there to help you to organize your work and parcel it out according to your preferred development methodology.

On the subject of TDD however, if you can't see how you might use it, then you either do not test your software enough or at all, or you have the most rigid testing regimen in place that it belies your need to unit test. TDD in itself is a very different way to write software in that you decide how you want your system to work, and before you implement it, you define how you will test it, then you write your code to pass the tests. It provides a means to streamline what you do, and shifts the focus from testing that your code works before you deliver it, to delivering working code confident that it already works.

If you feel a need to provide diagrams to show how your systems work, ask yourself where the value will be in producing and maintaining all of that additional documentation. Sure, it can be useful to document how a particularly difficult to define system works at a high level, but the more detailed your documentation gets, the harder it will be to maintain.

From the content of your post, it sounds to me as if you haven't really learned HOW to write software. If this is the case (you may be new or self-taught), then I would recommend at a minimum the following reading:

I'd also recommend looking into Behaviour-Driven Development as an alternative to TDD. Learn how to write software without adding piles of additional and wasteful red tape that you may not need.

What makes a software developer professional is how you manage your relationship with your customers, how you approach problems, and how innovatively you solve them. Also, it's how you work toward minimizing the clutter in your code, how thoroughly you test and how quickly you adapt to change, how you manage your workflow, and how you keep up with commonly accepted industry best practices. Well, there's obviously more to it than that, but it's a place to start.

  • Thanks. That's certainly important for common software projects and I'll have a look at the books, but it misses the question. It's not about refactoring, but about designing so that I don't need to refactor. I mentioned that TDD is dubious due the character of the data. Diagrams are not for documentation, but for planning. The code is perfectly clean and mostly bugless (and only 20000 lines), but data stream interactions are complex and sometime I find the need to refactor parts due to missing flexibility. And all only due to the lack of notation before writing the program (UML etc.) – Gerenuk Feb 13 '12 at 23:01

Not the answer you're looking for? Browse other questions tagged or ask your own question.