As a novice, how can I best overcome the complexity of larger projects?

Question

Currently, I'm a student and I can tackle all my assignments which only require, at most, 3-5 classes to implement. Now, I'm wanting to develop programs on a bigger scale but I am having a really difficult time figuring out how to design the program from a higher level.

The real world example I am working on is a group project to build an RSS reader. So far so good and I have a very basic prototype working that uses an RSS framework plus a few classes I created myself. Using the prototype as an example and the requirements document (well over 50 requirements, as of now) I need to build the actual program. I feel like I know a lot of the principles but I am not sure how to apply them to varying contexts such as this.

I'm hung up on how to break this simple RSS reader into multiple components, classes, etc and what those should even be. I would really like to know how I can/should go about wrapping my head around this project. I want to be able to step back and visualize the different pieces of the program so I have some structure before writing any code.

Extra kudos if you have a link to an article or blog that actually shows, by example, how you can take a concept and break it down at a high level. Sort of a "from concept to production" type article instead of just abstract information as available at sites like SourceMaking.

Answer 1

Designing the architecture of your project is the real assignment and they surely taught you the general principles, or they want you to learn those by yourself, practicing.

Anyway, a first approach could be:

1. Study your specs.
   1. Underline specific nouns in blue. They are probably your classes/objects/properties.
   2. Underline specific verbs in red. They are probably your main methods.
   3. Underline specific adjectives in green. They are probably your constraints.
2. Write each identified word in its own box (post-it).
3. Move those boxes around on a white dashboard.
   1. Related boxes should stay close to each other.
   2. Draw lines between boxes to see relationships.
   3. Add notes (stapled post-its) to the boxes.
4. At some point you'll realize that you need a new set of boxes.
   1. Draw them while looking at those on the dashboard.
   2. Add properties to boxes stapled to where they belong.
   3. Take a picture of the dashboard.
   4. Remove the old boxes and store them in a bag, just in case.
5. Repeat from 3 until you like what you have on the dashboard.

Answer 2

Writing programs on a larger scale is much like writing programs on a smaller scale. You just have more classes, that's all. To solve a larger problem, you break it into smaller pieces, and deal with the pieces individually using separate classes.

I think the best way to begin the learning process of tackling large projects is by studying design patterns., especially the structural patterns. Structural patterns are helpful for organizing collections of classes into larger organizational structure.

You can also study patterns like Model-View-Controller and multi-tier architecture.

Answer 3

Unless you work on one, it is fairly hard to imagine new complex systems. Given that, I have a 3 day plan to learn in a new place. This works, and if a new assignment/job you can start being productive quickly. Fundamentally, it is important to get the overall picture.

Learn three things, dedicating 1 day for each of the following

1. System Layout: It is very important to know the physical layout of the infrastructure. Learn about the different servers and their attributes, operating environments, et.c., essentially everything about the infrastructure.
2. Database Schema: Learn how the application's data is stored. Ask around and find out the most important data (table/view/relation...) This is the best way to learn about the business.
3. Overall Architecture: Learn about the important components of the systems and how they are glued together.

Answer 4

There are many ways to tackle this program, but whatever you do, don't begin without having a clear idea of where you are going. In your response to my comment, you say that

we have started doing all the requirements and documentation including various diagrams. I know what the program should do but I am not sure how to separate that into classes, methods, etc.

I would not worry too much about that. What you need to do as a first step is be really certain of the "flow" of your site. Think in terms of usability first, then sketch the background processes as if you were engineering a machine. Don't let classes and coding practices stop you at that stage.
When that part is over, you should be able, with the help of a few papers, to explain to coders and novices alike how it all works, both from the front-end and the algorithmic point of view. If you can't, then it's not clear enough. Get back to it.

When that part is done, 50% of the code design is done. From there on, there are many ways to go. For example, you could list all the functionalities (features) needed, all the data storages (database, filesystem, xml...), and all the front-end modules, and see what links with what the more naturally possible. From there, see what are the common features, and create mother classes and see what functions have to be added in subclasses. Pay special attention to what data enters a function and what data leaves it, and make sure related functions work well together.

I am personally a fan of getting real by 37 signals (it is an absolute must-read if you don't know it). What they do is design the interface first and then go from there. It helps to concentrate on what really matters, and check at each step that the flow is good.

My personal variation on that technique is: I write my end-code first. This helps me understand what I would want my classes to do. If my code gets too complex, then I change it on the fly (it doesn't do anything anyway, I haven't created a single class yet), and begin implementing features.
This is somewhat akin to test-driven development, albeit simpler to kick-start, because it needs 0 preparation.

And lastly, although good coding practices are a must, and I totally agree that they should be used at all costs, but don't make the beginner's mistake of being hindered by them. What matters is that the code works, above all. So keep in mind all the good stuff you learned, but don't stress too much and just dive in. Iterative design is the best pattern (at least for small groups). Build something that works, and optimize later (try to keep a minimal level of cleanliness though or you won't be able to optimize at all).

Hope it helps.

[edit] I forgot to mention: if you aren't already, using a subversion system can save a lot of headaches on the long run. Bitbucket offers private repositories, and it's fairly easy to set up mercurial on linux & windows (I don't know about apple, but I suppose it's just as easy).

Answer 5

• Habit. Having habits means that when you wonder what you did, you know you did it in the usual way so you can figure out what you did. Like where did you put that file.

• One Computer. Do all you work on one computer; preferably a notebook you can take places. That way every time you do something you have access to everything you ever did before.

• Subdirectories. Subdirectories are wonderful for keeping projects, source, versions, test data, etc.

• Copying. Ideally when you write a new program, half the code can be copied from an old program you wrote earlier.

• Testing. The three secrets of reliability are testing, testing, and testing.

• If things get too complex, consider a source code version control system. I don't use it because I'm always working along and I have no desire to back up to an earlier version of my source. But for projects inolving multiple people it is critical.

• Archives. Store "That program from three years ago.zip" on a CD.

Answer 6

I'm going to disagree with a lot that been written here, while writing an brain-dump in outline form with general features, scenarios etc that's about the extend of spec work you really have to do.

The easiest way to write a complex program is just writing it, and make sure you refactor as you go along. Whenever you add a new feature/aspect of the program, write a separate little app for just testing and experimenting with that part and then integrate it into the main app.

As your application grows you will start to see common themes, patterns and logical divisions in your code, refactor mercilessly to separate your app into these and reduce complexity and dependencies. The less dependencies the easier it is to understand one part of your code and it's also easier to write small test programs for that part of the code.

The main thing is always to stop at regular intervals and ask yourself, "Is the code getting too complicated? How can I fix it?" and then redo the code as you would have done it from scratch if you then knew what you now do.

Answer 7

I have suffered the same dilemma for too long I guess... Always wondering where to go next. I think what I was missing was some documentation e.g. Proper ERD, Data Flow Diagriam, and Design Documents like Use Cases/Stories || System Sequence Diagrams...

A software with proper requirement engineering and documentation does not suffer such a scenario in my personal opinion.

Have you prepared enough for your application before jumping in on coding part.... ???