It is one thing to say that agile methodologies are good in settings where requirements are poorly understood, or that significant novelty is involved. But should it be applied where outright innovation is required? If yes, how?

If what you are contemplating is unknown in the industry, or even thought to be impossible, it may be difficult to conceive of the user stories and related tasks. For example, would it have benefitted Albert Einstein (or a hypothetical employer he reports to) to have devised the theory of general relativity by breaking it down into epics, sprints, and tasks? If the answer is "yes," then what special accomodations should have been incorporated to help an agile approach work best with Einstein's way of achieving revolutionary insight?

To give a specific software example, imagine the year is 2008 and you would like to use WCF to provide COMET or "long-polling" type capabilities. All your research of "prior work" turns up nothing, and you even read an MSDN blogger say it isn't possible.

Again, what adjustments or "flavor" could be brought into the user stories and tasks to accomodate the inventiveness (or audacity?) of this endeaver? Or would it indeed be better to conclude the effort is so innovative (in 2008), it is best left as an undirected think-tank exercise?

The innovator who is operating under two-week sprints certainly does not want to be shot-down every time he abandons a dead-end task and starts working on a newly discovered task that was not envisioned when the sprint was defined. Likewise, when the sprint ends and no working code or working approaches are delivered, the innovator should not be shot down by management. There needs to be a way to label the effort as "success" even in these circumstances. In perhaps one or three more sprints of this kind of "dead end" pursuits, the innovator might finally hit on something that works.

How does agile allow management know that each sprint was "ok" despite the innovative set-backs? How is this managed so that the burn-down chart does not look preposterous?

  • 8
    @Liath: Innovation often needs to have the time to try out ideas without the pressure of having to show something every two weeks, i.e. at the end of the sprint. Short-term feedback often puts the focus on "show something, no matter what" (because it is always possible to fix it in a future sprint, if the customer is not satisfied) instead of "try to do it the way you think you should do it". "You show it when it is ready" instead of "You get it ready when you have to show it." – Giorgio Jun 30 '14 at 7:23
  • 4
    I think an offshoot question that can be made out of this question is: "Does time-boxing have its value in software research or highly innovative projects?", also, "Are there innovative/high-risk projects that do not benefit from time-boxing"? (I was reading this from an ad-hoc Google search: agile.conscires.com/2010/03/30/agile-for-research-projects ) – rwong Jun 30 '14 at 8:21
  • 3
  • 1
    This link, attributed to Xavier Amatriain, also appears to offer a complete package ("process") to applying Agile methodology in research projects. It is different from Scrum as we know it, but it goes very far in embracing Agile values and practices. – rwong Jun 30 '14 at 8:42
  • 2
    Innovation in software development is not easy regardless of methodology because people are taught (with good reason, I suppose) to stick to the things most people agree on. I think it's because software engineering is not very scientific, compared to other engineering disciplines, in which ideas are judged on their merits, not on their conformism. – Mike Dunlavey Jun 30 '14 at 13:04

The title question, where innovation refers to smaller-scale creative advancements in a team that is already doing well in Agile.

The best answer is summarized in this article about "Gold Card Days".

Summary (paraphrased, and with my own interpretations which may not reflect the author's intentions):

  • Developers can identify interesting (intellectually stimulating) stretch-goals that are work-related that they would like to work on.
  • These stretch goals, after being approved by the team (including the product owner), becomes "gold cards".
  • The team is encouraged to take out a day to work on these "gold cards".
    • Usually this happens to be on Fridays, so it becomes the "Gold Card Day".
  • With respect to Scrum, Gold Cards are scheduled and tracked just like any other backlog items; the team will need to demonstrate their results.

There are some other points (not in that article) in relation to applying "gold cards":

  • Do not let a single team member take all the fun. Every team member should be encouraged to spend some creative time by taking a "gold card day" once a while.
  • Along the same line, try to make a "gold card" a team effort (as opposed to a solo task), and exploit that task as a socializing (team-building) moment.

The substantial question, where innovation refers to research (months to years of gruesome work) which have a real risk of not finding any useful solution.

This earlier question, Which extreme-programming techniques are appropriate to use in a research environment? covers much of the ground of this question.

(Disclaimer: I wrote one of the answers for that question, though not the selected one.)

The summary is that software research work can be fast-paced; it requires its participants to prioritize based on new information (by absorbing discovered/learned ideas and synthesizing new ones). It gives the appearance of being "slow" only because it is "slow to show the fruits of success, and only if it was successful."

This question on Project Management Beta - What are the pros and cons of incorporating a project manager into a research team? - also covers the same grounds.

In spirit, yes.

As pointed out in mouviciel's answer, the spirit of software research is in line with the spirit of the Agile Manifesto. What I will argue next is whether high-risk research can be fit into Agile as a organization or management methodology ("Agile in practice").

In practice, you have to answer a few questions.
This list is not exhaustive...

We have to track back a little bit about how Agile Methodology comes into existence.

Agile Methodology is usually used when there is a sponsor of the project. Furthermore, the sponsor's willingness to fund the project is limited; it expects to see some software of usable (potentially shippable) quality being delivered on a regular basis after funding the project for some time.

The kind of research work in this question refers to "potentially non-solvable endeavors". In other words, the very nature of this type of work involves a risk that it can ultimately fail, despite all of the intentions and diligence of the people involved.

This is not a ScrumButt-style checklist.
This is more of a preflight checklist that predicts whether one is better off "Que Sera, Sera"

1. Transparency up-front. Is the sponsor of the project being told the truth about the riskiness nature of the project?

2. Willingness of the sponsor. Is the sponsor aware of the risk and willing to continue the funding?

The sponsor needs to have a risk acceptance that is higher than the typical business projects, or the typical Software / IT / Agile projects. Not every sponsor fits this criteria. If it doesn't fit, it would be good for a professional to back out from the project.

3. Transparency throughout the project. Is the sponsor being informed of the true status of the project regularly?

This is to thwart attempts to hide setbacks or looming failures in the project by misusing the time lapse between status updates.

4. Active participation of the sponsor. Is the sponsor interested in knowing the nitty-gritty details, the ups and downs, the promises and the limitations of each attempt?

The issue with software research is that there can be many false leads - both false positives (believing an approach would work but it ended up not successful), and false negatives (claiming something isn't possible, only to be disproved by someone else).

Agile projects allows the team (including the sponsors and stakeholders) to take calculated risks. "Calculated" means that the risk-takers are fully-informed. If the sponsor is not willing to learn the nitty-gritty details of the project, then the sponsor will not have complete information to calculate (judge) the risk on their own.

Even if a sponsor is willing to take the financial risk, if it is unwilling to also partake the decision-making risks (and accepts consequences to its own choices) then the sponsor is also unfit for such high-risk research projects.

5. Can the research team show (demonstrate) their progress in the form of running software, as opposed to presentation slides?

This question is appropriate for research projects where the end-result is expected to be running software. Presentation slides could be useful for explaining CS theories, but it could also be misused to hide setbacks in the software implementation (or the complete lack of). A software demo is intended to thwart such misuses.

6. Can the research team deliver a partially valuable software product, even if the sponsor decides to stop funding at any time in the project?

This question is only relevant on a case-by-case basis. Some research projects are incremental; they may have multiple milestones and deliverables. It requires a research team to prioritize their approaches to favor "lowest hanging fruits first", or the "least cost approach to demonstrate viability".

Some research projects are not incremental: to deliver a single, very specific technological breakthrough. It is a hit or miss. For this type of projects, the only incremental outcomes are the research work and prototyping, and maybe the academic publications. These "non-consumable" incremental deliverables are nonetheless valuable to some types of sponsors - namely, universities, research funding agencies, and big-name corporations hoping to build up academic goodwill.

However, research projects having such characteristics might also favor the "Cowboy coding" approach, as discussed further below. These are aptly termed "hacks", and they do occur in academia.

Because of the time-scale of most academic research, academic-style research funding is usually provided with a commitment of one or more years; medical research funding (academic and commercial) may be committed for even longer periods. On the other hand, typical commercially-funded research may be terminated without notice, or have their resources (manpower) being completely reassigned to other projects.

7. How does the research team measure on the scale of silo vs. cross-functional?

Some types of research team are highly-siloed. Often, this happens in "multi-disciplinary" projects - exactly one member from each discipline is involved. As a result, no one member can take over another member's task, not even very small ones, because their knowledge and skills do not overlap. The difficulty would also extend to communications and task definitions.

Extremely-siloed teams will still benefit from some Scrum rituals such as a daily stand-up meeting, but aside from the "ritual" there may not be much interaction going on. It would take a highly socializing agile coach to get the team into talking and building trust.

8. If an agile coach is present, does the coach prescribe short iteration cycles, time boxing, and providing time estimates?

Each of these agile practices pose difficulties for certain types of research projects. Nevertheless, it was reported that some expertly-skilled research groups were able to apply these practices in advanced research. Since there is no detail about how agile coaching occurs within these expert teams, we may not be able to know how each of these difficulties could be overcome.

9. Does the research team unanimously vote for adopting Solo development style over any other methodology?

Edited: an earlier version uses the phrase "cowboy coding", which alludes to the lack of professionalism. However, there is a difference between solo development and cowboy coding, and the circumstances in this checklist item may make solo development a legitimate choice.

This question shows that there are programmers who prefer to own a large chunk of development. If the research team is mainly comprised of this type of programmers, given that the team members' skill set are irreplaceable (referring to the previous point of skill silo), then the team members may have to be granted what they wish for, in exchange for their skills and labor.

The main difference between solo development and cowboy coding is that in solo development, one can adopt the practices listed in The Joel Test: 12 Steps to Better Code, such as the use of version control, build automation, and fixing bugs before adding new features.

Some circumstances will favor each member carrying out solo development, whereas some circumstances will favor cowboy coding.

Cowboy coding is favored if the end-goal is to "make a point", by showing that something is technologically possible. There is no requirements for delivery - nor quality - aside from a good presentation on the next DEF CON ®.

Final question. If the circumstances do not allow an Agile team to carry out groundbreaking research, then how do they make use of innovative technology?

Business as usual. Let other companies (or academics, individuals or teams of hackers, startups, etc.) tackle the hard problem first, and then buy/license the technology from them. The software industry has been running on these principles for many decades.

The emphasis on showing early working prototypes in Agile methodology compels a team to look for existing solutions first, which is a good thing because it may save the team from some redundant work.

| improve this answer | |

Back to Agile Manifesto:

  1. Individuals and interactions over processes and tools
  2. Working software over comprehensive documentation
  3. Customer collaboration over contract negotiation
  4. Responding to change over following a plan

Nothing in these values forbid innovation. Actually, innovation has a better nest with Agile than with Waterfall.

Nevertheless, it may happen that usual implementations of Agile put some constraints on a software project that limit innovation, such as deadlines (the timeframe of a sprint is a deadline) or cost. But this is not an issue with Agile, it is an issue with current work organizations.

| improve this answer | |
  • 3
    + I think you've got it. I think the problem is with books that tell people how to do it. (I've found it's very hard to write without making stuff up.) Our team follows "Agile" and what it means is endless meetings. One member simply said "Count me out. It's just the latest fad. If you don't need me, that's fine." – Mike Dunlavey Jun 30 '14 at 12:01
  • @MikeDunlavey - I like how your: Our team follows "Agile" resonates with: Responding to change over following a plan. – mouviciel Jun 30 '14 at 12:20
  • 1
    @mouviciel: I agree with your answer but then I do not understand what is really new about the agile values: I have been following at points 1, 2, 4 in all my projects long before I had even heard the word agile, and most of the people I worked with were doing the same. We called it common sense. So is the term "agile" just a new word for "don't be a slave of your process and use common sense"? On the other hand, the only real difference agile has brought to my work is more meetings and more rules to follow. – Giorgio Jun 30 '14 at 12:26
  • @Giorgio - Yes, this is the way I see it. The best waterfall projects I worked on were when the team leader favoured common sense among developers and told the customer a pretty "V-model/ISO9001/Huge Documentation" story. What is new with the Agile values lies in the credentials provided by the Manifesto's authors. – mouviciel Jun 30 '14 at 12:32
  • Agile was originally a manifesto about software development; it has been grown (or mutated) to address a wide variety of doing business. Meetings are a form of face-to-face communication, even though it would have been less effective than one-to-one communication because of politics and personal style. – rwong Jun 30 '14 at 13:15

I don't think it does. Agile is about eating chocolate elephants - taking a large task and breaking it down into manageable chunks that can not only be delivered, but are delivered regularly.

Research type projects do not fit into this - not unless your project can be broken down into small chunks that can be demonstrated every fortnight (or longer - nowhere does Agile say 2 weeks is the time your sprint has to take, best agile project I ever worked on had 6 week sprints)

I wouldn't try it though. I'd take the bits of agile tools that you think would work for you, and ignore those that don't. Too many people think Agile means "you must do everything the scrum tutorial says you must do and no discretion is allowed ever", that approach is very un-agile.

| improve this answer | |
  • 1
    Breaking down large tasks into small chunks is not an Agile thing. It has been used since the start of engineering back in ancient Mesopotamia and Egypt, and is widely used in Waterfall projects. If it is used in Agile projects, it is not because of its Agile nature but because of a mindset inherited from centuries of successful achievements. – mouviciel Jun 30 '14 at 8:06
  • @mouviciel: True, but agile forces you to break problems down to chunks that must fit into a single sprint. If they don't (like it is often the case in research projects), you are screwed... unless you make your sprints much longer. When you work on a complex research problem, you cannot foresee how long it will take to break it up into small enough pieces: having the small pieces means you have basically solved the problem already. – Giorgio Jun 30 '14 at 8:09
  • @rwong: Are there any agile processes other than Scrum that do not require as-early-as-possible feedback and short development cycles? – Giorgio Jun 30 '14 at 8:12
  • 4
    "Too many people think Agile means "you must do everything the scrum tutorial says you must do and no discretion is allowed ever", that approach is very un-agile.": True, my previous team was much more agile when we were doing waterfall: we were taking the pieces we needed and fitted them to our needs. Then came agile coaching and we had to work by the book: we lost most of our agility. ;-) – Giorgio Jun 30 '14 at 8:23
  • Let us continue this discussion in chat. – rwong Jun 30 '14 at 12:36

Not the answer you're looking for? Browse other questions tagged or ask your own question.