How to document code?
You already have a hint: look at how Java API is documented.
More generally, there is no unique set of rules which apply to every project. When I work on business-critical large-scale projects, the documentation has nothing to do with the one I would write for a small open source library, which, in turn, has nothing to do with the documentation of my medium-scale personal project.
Why many open source projects are not documented well?
Because most open source projects are made by people who contribute to those projects because it's fun. Most programmers and developers consider that writing documentation is not fun enough to be done for free.
Why many closed-source projects are not documented well?
Because it costs a huge amount of money to (1) write good documentation and to (2) maintain it.
The immediate cost (cost of writing the documentation) is clearly visible to the stakeholders: if your team asks to spend the next two months documenting the project, it's two additional months of salary to pay.
The long term cost (cost of maintaining the documentation) becomes noticeable pretty easy too to the managers, and is often the first target when they must lower the cost or shorten the delays. This causes an additional problem of outdated documentation which quickly becomes useless, and is extremely expensive to update.
The long term savings (savings from not having to waste a few days exploring the legacy code just to understand a basic thing which should have been documented years ago) are, on the other hand, difficult to measure, which confirms the feeling of some managers that writing and maintaining documentation is a waste of time.
What I often observe is that:
At the beginning, the team is willing to document a lot.
Over time, pressure of deadlines and lack of interest make it more and more difficult to maintain the documentation.
A few months later, a new person who joins the project practically can't use the documentation, because it doesn't correspond at all to the actual system.
Noticing that, management blames developers for not maintaining the documentation; developers ask to spend a few weeks updating it.
If management grants a few weeks for that, the cycle repeats.
If management refuses, based on previous experience, it only increases the bad experience, since the product lacks documentation, but a few months were spent writing and maintaining it.
Documentation should be a continuous process, just like testing. Spend a week simply coding a few thousands of LOC, and getting back to tests and documentation is very, very painful.
How to encourage the team to write documentation?
Similarly to the ways to encourage people to write clean code, to do regular refactoring, to use design patterns or to add enough unit tests.
Lead by example. If you write good documentation, your pairs might start doing it too.
Do systematic code reviews, including formal code reviews targeted at inspecting the documentation.
If some members of the team are particularly antipathetic to good documentation (or documentation at all), discuss the subject with them privately, to understand what are the impediments which prevent them from writing better documentation. If they blame the lack of time, you see the source of the problems.
Make the presence or the lack of documentation measurable for a few weeks or months, but don't focus on that. For example, you may measure the number of lines of comments per LOC, but don't make it a permanent measure, otherwise, developers will start writing long but meaningless comments just to get rid of low scores.
Use gamification. This comes together with the previous point.
Use positive/negative reinforcement.
(See the comment by SJuan76) Treat the lack of comments as errors. For example, in Visual Studio, you can check an option to generate XML documentation. If you also check that all warnings are treated as errors, the lack of a comment at the top of a class or a method will halt the compilation.
As for the three previous points, this one should be used with caution. I used it for a while with a particularly tough team of beginner programmers, and it ended up with StyleCop-compliant comments like that:
/// <summary>
/// Gets or sets the PrimaryHandling.
/// </summary>
public Workflow PrimaryHandling { get; set; }
which were, hm..., not particularly helpful.
Remember: nothing automated can help you pinpoint bad comments when programmers wants to screw with you. Only code reviews and other human tasks will help.
Are there any good examples of when minimal or no documentation is needed?
Documentation explaining the architecture and the design is not needed:
For a prototype,
For a personal project written in a few hours to accomplish a task, while being pretty sure this project won't be maintained any longer,
For any project where it is obvious, given the small size of it, coupled with the particularly clean code, that you will spend more time writing documentation than all the future maintainers exploring the code.
In-code documentation (code comments) is not needed according to some developers when the code is self-documenting. For them, the presence of comments is, except in the rare cases, not a good sign, but a sign that the code wasn't refactored enough to be clear without the need for comments.
I feel that we should have a documentation review after a project is delivered.
If your project is delivered at least once per week, it's the way to go. If your project is not agile and is delivered at the intervals of six months, then do more regular reviews.
