When should "Epics" be defined in the "Agile" process?

Question

So I understand what an epic is, but since i've never seen "working" agile in the field it's tough to understand "when" things happen in the agile process.

To me, I know you gather requirements at the beginning (obviously), but do you FIRST create epics out of these? and then split them up into individual user stories for the backlog for each sprint?

or are epics created during each sprint per sprint? To me it makes more sense for epics to be created during the discovery phase or right after requirements are gathered, but maybe I just have a misunderstanding of the discovery phase.

Answer 1

When you gather requirements, you typically interview people, read and evaluate business or technical papers, or you look what competing software already provides or misses. You will get lots of ideas, on lots of different levels of abstractions. If among these ideas and input you find clean, short and precise problem statements, there is no need to inflate these artificially to an "epic". You use an "epic" if you cannot immediately break down the idea for a requirement to a smaller sub-problem, and you use a "user-story" when your input can be immediately written down in form of a "small user story". And thats it.

This is mostly independent from being in a point in time where no software is written so far (what you call "discovery phase"), or a point in time where you already have a working product and can describe any new requirement in form of a change to the existing software (which should be the 99% case in agile). Indeed, when you have a working product of a certain size, it will become certainly easier to describe several requirements in detail by referring to the existing product and writing a small user story which describes the requirement in form of a change request. Nevertheless, even if the product has already lot of features, expect users to come up with new ideas at any point in time which require an "epic" as the appropriate form of description.

Note also that the typical "life cycle of an epic" is that it will be broken top down to user stories which can be developed individually (and not the other way round), because an epic is typically too big to be developed in one sprint. There might be cases where lots of individual, detailed "user stories" inspire an "epic", because one gets a "bigger picture" from a collection of ideas - but it won't make sense to throw the related user stories away then, because you will need them either for the detailed sprint planning at a later point in time.

Answer 2

Short answer: epics can always be defined while the product is alive

Long answer: it depends a little bit on how you use epics and develop your product roadmap. I've seen basically 2 ways of usage so far.

1. Epics as "big user stories"
2. Epics as organizational layer above user stories

Epics as "big user stories" is what is mostly used when people talk about epics. Whenever you get new requirements and they are too complex or not concise enough you can pack them into an epic and into the backlog to start working. During refinement sessions you drill down the requirements to nicely implementable user stories and in the end the epic vanishes from the backlog, only the let's say 12 user stories that were created from it remain. This happens all the time during a product is alive. Every new, big feature request will lead to a new epic.

Epics as organizational layer above user stories is what I have seen often in the places where agile meets classical corporate waterfall worlds. It is then just a layer of abstraction on the user stories. A little bit like categories that are often defined in the beginning of a product development (e.g. during discovery phase). For example "new customer", "printing", etc. There might be some new categories (=epics) coming in the future, but only if you are changing your product roadmap to include something completely new in the future.

If you can choose the way you work, go for #1. I included #2 here as an example on how epic is used a lot in practice, not as recommendable option.

Answer 3

I would say epics should come before stories (assuming you have some semi-long term goals). The idea of the epic in my opinion is to track larger projects. In waterfall the dev cycle would often be months long, most agile teams run 2-week sprints. You can't do anything hugely significant in 2-weeks, so how do you track these bigger ideas? With epics.

On my team, epics are just a tag. It's an idea, nothing more. Take for example implementing a new account management service, something that might take several months for one or two scrum teams to complete all the functionality the market is demanding. In that case we would create an epic tag called account-management-service and tag all our stories related to it with that. Really this just helps to organize things. With the big picture idea in mind you can begin writing these smaller user stories that can and should be completed in 2 weeks or less.

In many cases epics aren't necessary. If I already have a working REST API and consumers of it are just requesting a couple new endpoints to deliver data in a more efficient format for example, there's not much need for an epic here. We write a story and it gets done in a sprint and it's a relatively independent effort. If your team has a vision in place that will take months to implement, that would be a good time to bucket your stories into 'epics' so that you have more visibility into the progress towards those larger goals, though they shouldn't really effect story pointing, planning or anything else in your day to day.

Other orgs may use these differently, to me this is the most sensible use for epics though.