Looking for an example of how a software project can be managed/deployed

Question

My company is evaluating adopting off-the-shelf ALM products to aid in our development lifecycle; we currently use our own homegrown solutions to manage requirements gathering, specification documentation, testing, etc. One of the issues I am having is understanding how to move code between stages of development. We have what we call a pipeline, which consists of particular stops:

[Source] -> [QC] -> [Production]

At the first stop, the developer works out a solution to some requested change and performs individual testing. When that process is complete (and peer review has been performed), our ALM system physically moves the affected programs from the [Source] runtime environment to the [QC] runtime environment. This movement of code is triggered by advancing the status of the change request to match the stage of the pipeline.

I have been searching the internet for a few days trying to find how the process is accomplished elsewhere -- I have read a bit about builds, automated testing, various ALM products, etc. but nowhere does any of this state how builds interact with initial change requests, what the triggers are, how dependencies are managed, how the various forms of testing are accommodated (e.g. unit testing, integration testing, regression testing), etc.

Can anyone point me to any resources detailing specific workflows or attempt to explain (generically) how a change could/should be tracked and moved though the development lifecycle? I'd be very appreciative.

Note: I've cleaned up the question to hopefully make it easier to understand. Also, I found another question (which I can't find now) that referenced this book, which sounds like it might be exactly what I am looking for -- not sure if I want to shell out the cash for it, though.

Additional details per @GlenH7's suggestion:

Our products are built with a proprietary language and set of translation tools -- the ultimately runtime environment is similar to a web site architecture, in that we have a bunch of 'code' files sitting out on a server, which our client app accesses on-demand (not unlike a GET request) and then interprets to provide the user experience. We sell an integrated product which is made up of a set of modules (most of which are optional), each module has a team of ~4 programmers and 2 QC testers devoted to the development/bug maintenance. There are around 30 modules per product line (and 3 product lines) -- some teams maintain multiple products lines for their given module. In total, we have 700 employees devoted to development and ~200 programmers. Certainly adopting any new processes or systems is going to involve buy-in, etc. but I'm tasked with simply understanding how the code will be managed so that we can speak intelligently about things. My first task was to learn how to adopt version control into our process (we do not yet use formal version control for most of our product lines, we rely instead of keeping around multiple copies of the entire codebase).

We follow the waterfall method pretty closely (which sucks, but the company has been producing software since 1969, so it was probably the natural fit back then and until now, there has been no need to change anything). Some folks are intrigued by the various offerings of the agile methodologies, but as has been noted by @Chad below, we're likely going to need to take incremental steps towards any such end goals (though if we adopt an ALM system, it might be the time to just force a whole new way to do things on everyone).

Now for a specific example of our current process, let's say we have a product called HIS, which has an MIS and ITS module (what these are/could be is not important -- think of them as just a collection of files) -- in our current process, if someone requests a change to ITS, a coder will modify some files (effectively) on the development server. When that change needs to be tested, transitioning the status of the change request from 'development needed' to 'qc needed' will trigger our tools to copy the affected files from the development server to the QC server. Now when QC testers signon to the QC testing environment, they will be able to see the effect of the changes. Once they signoff on the change, advancing the status of the request to 'production ready', our tools again copy the code files to the next stage. Once signed off there, a final copy is done to the holding location used for deliveries to customers.

We manage delivery of our releases by a major release number (e.g. 6.0) followed by a service release designation (e.g. 5) and a 'priority pack' number (e.g. 1) -- customers take delivery of priority packs, e.g. 6.0.5.1. As stated above, we simply push code from place to place on multiple servers, so we end up with a project folder for each major release and service release throughout the pipeline and end up with a final resting place for each of the ppacks, e.g.:

dev6.0
qc6.0
production6.0

dev6.0.5
qc6.0.5
production6.0.5

ship6.0.5.1

I was thinking that under version control, we could simulate this kind of scheme as:

trunk
branches
--6.0.5
tags
--6.0.5.1

but that seems to cover only half of the locations, i.e. where do the qc and production sources live? Would I create a separate trunk for each major and minor release of the product? Or would I rely on something outside of version control (e.g. the ALM system) to track the revision(s) interesting for testing done at the QC and production stops? We'd need a runtime environment for each, still, but that could be 'built' on-demand as bug fixes and features are made available -- which reinforces the idea that I need not keep any special 'copy' of these stages under version control (they are simply references to revision history of the trunk or branch).

Another thing that has thrown me for a loop as that a colleague is pretty adamant that you should not persist anything in branches, i.e. the branch folders should be ephemeral and used only during large changes that can't be committed directly to the trunk (with the expectation that the branch will eventually be merged into the trunk). In his scheme, the minor releases would necessitate a separate top-level with it's own set of trunk, branches and tags, e.g.:

product6.0
trunk
branches
--silly_new_feature
tags
--6.0.5

product6.0.5
trunk
branches
--crazy_bug_fix
tags
--6.0.5.1

(sorry for the delay on adding these details -- got pulled away to meetings all day)

Answer 1

The approach I have found works well is two build environments: development and testing. Development build occur frequently and allow integration issues to be worked out and changes (including features) to be tested under the standard build.

Testing builds are triggered on demand, and if accepted the build will be promoted to production. Testing builds may take place on a separate branch, especially near a planned release. Changes on the branch will need to be integrated back into development.

There are some assumptions taken with this approach:

• Mainline development is done on trunk. (Feature development is either for the next release, or can be released in an inactive state.)
• Any environment specific information is retrieved from the environment. When changing environments no builds should be required. Various mechanisms can be used.
• Releases consist of a coordinated set of feature and bug changes. (If there are dependencies between features or bug fixes, pulling one can cause a cascade of changes.)
• A separate branch is created for managing fixes to a released (or releasing) version. This may include an additional development build environment.
• Build tagging is done on testing builds. This simplifies loading that code if changes are required, or code needs to be investigated.

I haven't used any ALM software, as it tends to be low on the budget wishlist. The above approach is fairly easy to implement using standard build and configuration management tools.

One thing I have noticed with ALM is that retirement is often not well managed. It is rare to see temporary features or bug fixed actually removed from the code. This may result in part from change tracking closing the feature on implementation. It may be a good idea to add a retirement change ticket when appropriate.

EDIT: I have seen a number of failures of tested code that requires a build to move into a new environment. It is difficult to ensure only the environment configuration is included in the build and not other data.

Things that change between environments include and how I have provided them:

• Database connections: The solution I usually use is to have the container provide a named connection pool. Different environments have different configurations, but this is transparent to the deployed application.
• Directories: When appropriate, using relative directory paths works well. Using environment provided variables as part of the path, covers most other cases.

• Properties: These can be used to help configure database connections, directories, and a whole variety of other settings. There are a number of alternatives which can be used (possibly in concert):

  • Configure the container with the appropriate properties and read the properties from there.
  • Provide a properties file in a directory relative to the environments directory.
  • Read properties from a database connection.

In some implementations, one of the properties specifies the location. This can be displayed in the the interface, to enable quick determination of which environment is being used. Build version may also be shown, but that should be part of the build artifacts, and not the environment specification.

Answer 2

Start here: Software Development Process
In particular, look at the Iterative approach and the Agile approach. Also check out Unified Process for bonus reading. UP gets a bad rap when folks think everything has to be included. That's not the way it works - you only pick what you need from UP.

Then go here: Systems Development Life Cycle

Most of your questions should be answered from those two resources.
You'll want to make sure your ALM software has or can integrate with issues tracking as that's a critical component of SDLC. Here's a handy comparison link, ymmv.

As far as QA doing its thing, what I've seen work best is this: Dev has a feature that they work against. Once the feature is "code complete," it's released to QA and is marked as "in test" or "done." Bugs / issues can then be opened up against the feature that was released. As the bugs are fixed, they too can go into "in test" or "done." Feature ships when QA says its done or mgmt says it's good enough.

Update #1 I think your first priority is to decide what version control system you want to use. For the environment you described, (almost) anything will be better than nothing. Git and Mercurial are two very solid options to consider. I'm not as familiar with Git, but to my understanding the feature-sets are very similar.
Hg Init
Joel on DVCS
SO conversation on DVCS

Next up, start looking into branching patterns and identify what's going to work best for your organization.

Atwood's thoughts

The SVN book off of red-bean is well worth the time to read through.

Your (D)VCS needs to be decided prior to your ALM software. A fairly common pattern for your case would be to have branches / lines / (insert your term here) for dev, test, and prod releases. You can additional additional lines based upon your major versions. From your description, I think tags / equivalent within the main line for the version would be fine for your service releases and patches. It can be a complicated setup, but given the number of developers and different activities you have going on, I think it will be required.

Your coworker is partially correct about branches, nowhere near enough correct to be adamant about it. Some models only use branches for ephemeral work. Generally, that only works with smaller shops that don't have multiple releases running at the same time. You could appease him by having multiple trunks based upon versions, but that's a serious PITA. Spolsky's Hg Init covers some basic patterns that address your situation including a workflow.

Mercurial (and by association git) is the closest (D)VCS that I've seen that replicates the release environment for a large development team. Back in the day, we used a proprietary product that was release oriented and I've yet to find anything that truly recreates that. Hg + some integrated bug tracking will put you well on the way to automating your release life cycle.

Start with Hg Init; research the other options; research branching patterns; revisit the Hg Init tutorial. If nothing else, you'll have a boodle more terms to plug into SO and P.SE to continue your research with.

Answer 3

What you need to do is define your software development release process.

You can start by simply formalizing your existing process. I would recommend getting your developers input on the process so that they feel like they had a voice in the process. Have a regular review of the process with lessons learned. What is working? Where are our pain points? What can we do to improve?

Make gradual improvements in the process with your teams buy in. They are professionals and will probably understand the value in the controls being in place.

I suspect that book you list in your question will have a solution that will be beyond your reach for a first iteration of your process. That does not mean you can not try and steer your team towards that solution, but proven, effective, and consistent process compliance does not happen overnight. An iterative approach tends to work better, especially if you do not have any one with experience in the process you are trying to cram down their throats. There are quite a few methodologies and write ups on the SLDC on the internet. I would start with the free write-ups before I spent money on an advanced system that may or may not ever work in your environment.