I've been developing web apps for a while now and it is standard practice in our team to use agile development techniques and principles to implement the software.

Recently, I've also become involved in Machine Learning and Natural Language Processing. I heard people primarily use Matlab for developing ML and NLP algorithms. Does agile development have a place there or is that skill completely redundant?

In other words, when you develop ML and NLP algorithms as a job, do you use agile development in the process?

  • What? Are you suggesting that Agile development doesn't apply to certain programming languages? What are you asking?
    – S.Lott
    Commented Dec 29, 2011 at 1:21
  • I'm asking that when you develop ML and NLP algorithms, do you use agile development practices or not?
    – siamii
    Commented Dec 29, 2011 at 1:26
  • 2
    Designing algorithms and developing applications are two very different things. Commented Dec 29, 2011 at 1:56
  • 2
    Why would the the problem domain change the project management discipline? I'm unclear on what connection you're claiming exists between problem domain, language and project management. Can you clarify why Agile doesn't apply to this domain or programming language?
    – S.Lott
    Commented Dec 29, 2011 at 2:01
  • @S.Lott I can't tell you because I haven't worked as ML dev yet. In web app dev, you capture features, prioritize, pair program, write tests and do sprints with fast iteration. You have a prototype working very fast. On the other hand, if I wanted to join as a ML / NLP expert to a hedge fund to improve their automated trading system strategy, would I be able to use my experience of developing web applications with agile methodology? I imagine it's quite different than developing business web apps, since you don't have features, just an aim to improve prediction or something similar.
    – siamii
    Commented Dec 29, 2011 at 3:53

1 Answer 1


Maching Learning and Natural Language Processing is somewhat data-driven. Without a continuous supply of high-quality data (which must be re-captured whenever new criteria are added), the software development may miss its intended target.

The customer and product owner may devote a bigger fraction of their time toward test data collection.

The adaptations will depend on:

  1. The balance of time allocated toward "spike" versus "implementation"
    • spike: open-ended research / exploratory prototyping (where the benefits are possible but not certain, and where priorities are fast-changing)
    • implementation (where the benefits and costs are somewhat more predictable, but each task unit will take much longer to finish).
  2. Average size of a task unit. How long does a spike take? How long does an implementation take? Unlike typical application development, partial algorithm implementations are usually not runnable.
  3. Whether "canned algorithms" i.e. existing libraries / algorithm packages are available.
    • When these are available, time spent on "implementation" is reduced (because they're already written) and thus more time will be spent on "spikes".

There will be two feedback loops:

  1. In each iteration, customer and product owner collect/update data periodically (and with better quality / newer criteria) based on business need and developers' feedback.
  2. In each iteration, developers try to improve algorithm quality based on the data available, and the algorithm is packaged into a usable system and delivered at the end of each iteration. Customer and product owner should be allowed to take the beta system elsewhere, redistribute them, etc.

Thus, we see that "data" replaces "features" as the main definition of progress.

Because of the increased importance of "research / spike" in ML/NLP development, there is a need for a more organized approach to spike - something you may have already learned from any graduate research teams. Spikes are to be treated as "mini-tasks", taking from hours to days. Implementation of one algorithm suite will take longer, in some cases weeks. Because of task size differences, spikes and implementations are to be prioritized separately. Implementations are costly, something to be avoided if possible. This is one reason for using canned algorithms / existing libraries.

The scrummaster will need to constantly remind everyone to: (1) note down every observation, including "passing thoughts" and hypotheses, and exchange notes often (daily). (2) Spend more time on spikes (3) use existing libraries as much as possible (4) don't worry about execution time - this can be optimized later.

If you do decide to implement something that's missing in libraries, do it with good quality.

Daily activities:

  • Reprioritize spikes (short tasks / hours - days) and exchange research notes.
    • De-prioritize spikes aggressively if yesterday's result don't seem promising.
  • Everyone must commit a fraction of time on implementation (long tasks / weeks), otherwise nobody would be working on them because they tend to be more boring on tasks.

Sprint activities:

  • Demo, presentation of beta software
  • New data collection
  • Retrospect: data collection criteria / new measures of quality, algorithm satisfaction, balance between spikes and implementations

About the note on deferring optimization: The thought-to-code ratio is much higher in ML/NLP than in business software. Thus, once you have a working idea, rewriting the algorithm for an ML/NLP application is easier than rewriting a business software. This means it is easier to get rid of inefficiencies inheritant in the architecture (that is, in the worst case, simply do a rewrite.)

(All editors are welcome to rearrange (re-order) my points.)

  • do you write unit tests before you implement machine learning code in Matlab?
    – siamii
    Commented Jan 2, 2012 at 21:55
  • 1
    @bizso09: perhaps i'm not qualified to answer that question because I do not implement self-learning code. Instead, I design and implement algorithms that are inspired from clues used by human visual system. To answer that question, we first need to qualify the types of "unit tests". Please refer to Unit Testing Machine Learning Code
    – rwong
    Commented Jan 4, 2012 at 5:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.