The title question, where innovation refers to smaller-scale creative advancements in a team that is already doing well in Agile.
The best answer is summarized in this article about "Gold Card Days".
Summary (paraphrased, and with my own interpretations which may not reflect the author's intentions):
- Developers can identify interesting (intellectually stimulating) stretch-goals that are work-related that they would like to work on.
- These stretch goals, after being approved by the team (including the product owner), becomes "gold cards".
- The team is encouraged to take out a day to work on these "gold cards".
- Usually this happens to be on Fridays, so it becomes the "Gold Card Day".
- With respect to Scrum, Gold Cards are scheduled and tracked just like any other backlog items; the team will need to demonstrate their results.
There are some other points (not in that article) in relation to applying "gold cards":
- Do not let a single team member take all the fun. Every team member should be encouraged to spend some creative time by taking a "gold card day" once a while.
- Along the same line, try to make a "gold card" a team effort (as opposed to a solo task), and exploit that task as a socializing (team-building) moment.
The substantial question, where innovation refers to research (months to years of gruesome work) which have a real risk of not finding any useful solution.
This earlier question, Which extreme-programming techniques are appropriate to use in a research environment? covers much of the ground of this question.
(Disclaimer: I wrote one of the answers for that question, though not the selected one.)
The summary is that software research work can be fast-paced; it requires its participants to prioritize based on new information (by absorbing discovered/learned ideas and synthesizing new ones). It gives the appearance of being "slow" only because it is "slow to show the fruits of success, and only if it was successful."
This question on Project Management Beta - What are the pros and cons of incorporating a project manager into a research team? - also covers the same grounds.
In spirit, yes.
As pointed out in mouviciel's answer, the spirit of software research is in line with the spirit of the Agile Manifesto. What I will argue next is whether high-risk research can be fit into Agile as a organization or management methodology ("Agile in practice").
In practice, you have to answer a few questions.
This list is not exhaustive...
We have to track back a little bit about how Agile Methodology comes into existence.
Agile Methodology is usually used when there is a sponsor of the project. Furthermore, the sponsor's willingness to fund the project is limited; it expects to see some software of usable (potentially shippable) quality being delivered on a regular basis after funding the project for some time.
The kind of research work in this question refers to "potentially non-solvable endeavors". In other words, the very nature of this type of work involves a risk that it can ultimately fail, despite all of the intentions and diligence of the people involved.
This is not a ScrumButt-style checklist.
This is more of a preflight checklist that predicts whether one is better off "Que Sera, Sera"
1. Transparency up-front. Is the sponsor of the project being told the truth about the riskiness nature of the project?
2. Willingness of the sponsor. Is the sponsor aware of the risk and willing to continue the funding?
The sponsor needs to have a risk acceptance that is higher than the typical business projects, or the typical Software / IT / Agile projects. Not every sponsor fits this criteria. If it doesn't fit, it would be good for a professional to back out from the project.
3. Transparency throughout the project. Is the sponsor being informed of the true status of the project regularly?
This is to thwart attempts to hide setbacks or looming failures in the project by misusing the time lapse between status updates.
4. Active participation of the sponsor. Is the sponsor interested in knowing the nitty-gritty details, the ups and downs, the promises and the limitations of each attempt?
The issue with software research is that there can be many false leads - both false positives (believing an approach would work but it ended up not successful), and false negatives (claiming something isn't possible, only to be disproved by someone else).
Agile projects allows the team (including the sponsors and stakeholders) to take calculated risks. "Calculated" means that the risk-takers are fully-informed. If the sponsor is not willing to learn the nitty-gritty details of the project, then the sponsor will not have complete information to calculate (judge) the risk on their own.
Even if a sponsor is willing to take the financial risk, if it is unwilling to also partake the decision-making risks (and accepts consequences to its own choices) then the sponsor is also unfit for such high-risk research projects.
5. Can the research team show (demonstrate) their progress in the form of running software, as opposed to presentation slides?
This question is appropriate for research projects where the end-result is expected to be running software. Presentation slides could be useful for explaining CS theories, but it could also be misused to hide setbacks in the software implementation (or the complete lack of). A software demo is intended to thwart such misuses.
6. Can the research team deliver a partially valuable software product, even if the sponsor decides to stop funding at any time in the project?
This question is only relevant on a case-by-case basis. Some research projects are incremental; they may have multiple milestones and deliverables. It requires a research team to prioritize their approaches to favor "lowest hanging fruits first", or the "least cost approach to demonstrate viability".
Some research projects are not incremental: to deliver a single, very specific technological breakthrough. It is a hit or miss. For this type of projects, the only incremental outcomes are the research work and prototyping, and maybe the academic publications. These "non-consumable" incremental deliverables are nonetheless valuable to some types of sponsors - namely, universities, research funding agencies, and big-name corporations hoping to build up academic goodwill.
However, research projects having such characteristics might also favor the "Cowboy coding" approach, as discussed further below. These are aptly termed "hacks", and they do occur in academia.
Because of the time-scale of most academic research, academic-style research funding is usually provided with a commitment of one or more years; medical research funding (academic and commercial) may be committed for even longer periods. On the other hand, typical commercially-funded research may be terminated without notice, or have their resources (manpower) being completely reassigned to other projects.
7. How does the research team measure on the scale of silo vs. cross-functional?
Some types of research team are highly-siloed. Often, this happens in "multi-disciplinary" projects - exactly one member from each discipline is involved. As a result, no one member can take over another member's task, not even very small ones, because their knowledge and skills do not overlap. The difficulty would also extend to communications and task definitions.
Extremely-siloed teams will still benefit from some Scrum rituals such as a daily stand-up meeting, but aside from the "ritual" there may not be much interaction going on. It would take a highly socializing agile coach to get the team into talking and building trust.
8. If an agile coach is present, does the coach prescribe short iteration cycles, time boxing, and providing time estimates?
Each of these agile practices pose difficulties for certain types of research projects. Nevertheless, it was reported that some expertly-skilled research groups were able to apply these practices in advanced research. Since there is no detail about how agile coaching occurs within these expert teams, we may not be able to know how each of these difficulties could be overcome.
9. Does the research team unanimously vote for adopting Solo development style over any other methodology?
Edited: an earlier version uses the phrase "cowboy coding", which alludes to the lack of professionalism. However, there is a difference between solo development and cowboy coding, and the circumstances in this checklist item may make solo development a legitimate choice.
This question shows that there are programmers who prefer to own a large chunk of development. If the research team is mainly comprised of this type of programmers, given that the team members' skill set are irreplaceable (referring to the previous point of skill silo), then the team members may have to be granted what they wish for, in exchange for their skills and labor.
The main difference between solo development and cowboy coding is that in solo development, one can adopt the practices listed in The Joel Test: 12 Steps to Better Code, such as the use of version control, build automation, and fixing bugs before adding new features.
Some circumstances will favor each member carrying out solo development, whereas some circumstances will favor cowboy coding.
Cowboy coding is favored if the end-goal is to "make a point", by showing that something is technologically possible. There is no requirements for delivery - nor quality - aside from a good presentation on the next DEF CON ®.
Final question. If the circumstances do not allow an Agile team to carry out groundbreaking research, then how do they make use of innovative technology?
Business as usual. Let other companies (or academics, individuals or teams of hackers, startups, etc.) tackle the hard problem first, and then buy/license the technology from them. The software industry has been running on these principles for many decades.
The emphasis on showing early working prototypes in Agile methodology compels a team to look for existing solutions first, which is a good thing because it may save the team from some redundant work.
