What's the best way to do a Proof of Concept shootout?

Question

In preparation for a new release of software our company maintains, I've been working on what I believe to be a really good approach to solving our scalability issues. I have every intention of putting together a proof of concept to validate the design on paper will actually do what I want. When I briefed it to the team, the boss had a counter proposal, inspired in part by the way I described the problem areas. The boss also accepted my proposal to do two proofs of concept to evaluate the alternatives.

So, what's the best way to work through the proof of concept shoot out? We have both objective and subjective criteria that we are using to evaluate the solutions. I'd like to make sure we are comparing apples to apples with these fairly different approaches.

• We have requirements for throughput and size. In short, we know we need to process a certain number of objects per second and maintain that rate for an hour.
• We need to evaluate scalability (both by adding more cores and by increasing the number of objects)
• We need to evaluate ease of development (subjective)
• We need to evaluate how easy it is to understand the algorithm (subjective)

I have my theory about which way things will lean, but I don't want that to influence my results. Any input on how to maintain objectivity in this process, and things I might need to consider would be greatly appreciated.

Answer 1

Generally what we do is...

• set a deadline like 1 month
• use your requirements that you listed and code it up.

• have some moderator who isn't writing either program control the requirements and doesn't see them think of a hidden requirement that should be doable in a short amount of time.

• When the rest of the team looks over the programs to evaluate them THEY have to add in the requirement to each system so they get a feel for both of them. After the team dig into the code and add in the missing requirement they should have a reasonable feel for the codebases and pick which one they would rather develop off of.

  • then take the best remaining peices of the 'loser' and incorperate them into the winning architecture.

Answer 2

For the subjective items come up with a numerical rating of some time and try to obtain unbiased feedback. Example: for "Understanding the algorithm" have a programmer who did not write either look at both and rank each against each other.

You can also factor in objective measures on the code like the codes "Complexity", there are some tools to measure that based on number of control statements etc.

Take your ranking in each category and sum them up to a "Total Score" for each approach.

Answer 3

how to maintain objectivity in this process

You have only one objective criteria. The throughput.

Everything is subjective. You can't be "objective". All you can do is be "fair". World of difference.

The final decision is always political. As long as all the available information was provided; you've done all you can do.

Don't stress out over trying to make the perfect ("objective") point. What you see as right or best may simply be overturned by some ridiculous excuse like "the team doesn't have the required skills for your proposed solution".

Just build the demos. Run them. Be prepared for random decision-making. The very best you can hope for is informed and fair. You can't get to "objective" very easily.