Somebody explained to me that since in agile development, policy and the application logic should be more important than details such as persistence method, persistence decision should be taken at the end. So it might be a good idea to start with simpler persistence such as flat files, until we reach the point the weakness of this method become apparent, and only then we change the persistence, e.g. by using relational database.

Is this true, or did I misunderstood the concept? Is this how agile team usually build application? What are the rationales for it and when I should not take this approach?

  • 1
    Persistence logic is not part of minor details, or less important. That must be the first decision. You want ACID properties for your persistence structure. That decision can not be kept pending.
    – Manoj R
    Feb 7, 2013 at 13:03

9 Answers 9


The concept being conveyed is something which is definitely part of agile and relevant, the idea of pushing things off to the last responsible moment.

However the example taken actually relies on a completely false assumption to begin with:

that it is easier/less work to implement a flat-file database than use an RDBMS. -- Often completely false

The example should be: The persistence layer is kept to the simplest possible implementation until a decision is made that it is inadequate.

For many development teams, getting a database stood up to do this is a matter of an hour or two (or 15 minutes for small database with ORM) while a flat-file database that they don't keep needing to meddle with may be an enormous pain and annoyance because they have to hand-write all the seek and data-table construction type code manually, when a database may be as simple as creating a table in a UI, adding a few columns, and then having an ORM generate everything else you need.

Furthermore, if you don't know your persistence layer to begin with it is an even more appropriate act to start with a common RDBMS that your team knows well, as to make the change later from flat-file to RDBMS is much larger than later changing from one RDBMS to another. There are many tools for converting from most any common RDBMS to other such ones, and tips and such because it is a well-travelled path. There are extremely few tools for converting from a flat-file to any given RDBMS where your flat-file has some proprietary format that tooling hasn't previously been written for aside from your own libraries.

In summary: The concept is correct and accurate, but the example is based on a terribly large and often(almost always) inaccurate assumption.

  • 6
    And your terribly large assumption is that they have to hand-write all the seek and data-table construction type code manually! :-) Usually when you use a flat file you start by using your language's built-in serialization format (e.g. Marshal in Ruby, or NSKeyedArchiver in Cocoa/Objective-C) and just dump out some existing internal objects. As long as your app doesn't have to reload too often, and as long as you get a handle on schema changes across app versions, this technique can work remarkably well for a long time, especially during development. Feb 6, 2013 at 22:36
  • @AlexChaffee fair point, but either way you need to write some code around this stuff one way or another and modern ORMs make doing such things with an RDBMS or NoSQL for that matter fairly equivalent in triviality, where the difference in impact on the team is based more in the teams skillset than anything, I just think it's a bad example to illustrate the point it's trying to for this reason. Personally I've used MSSQL for 13 years but put on the spot would spin up MongoDB for persistence because of simplicity to not let it effect the real goal of the project until ACID mattered. Feb 6, 2013 at 22:39

Since you know you will be using a DB, there's not much point in writing code to handle flat files. You might get through a few iterations using some read-only CSVs but you'll quickly find yourself writing code that you know you will throw away.

One thing you can do to simplify the initial complexity by using something like SQLite (a library that gives you a serverless SQL database that's stored in a file). It's got a flexible type system so you don't have to seriously commit to a schema, no server to configure/connect to & rebuilding your DB is as simple as deleting a file. It also allows you to simply include the DB image along with the code in version control if needed.

  • 4
    Seems like you got a downvote from the Flat File Society.
    – JeffO
    Feb 6, 2013 at 18:04
  • @JeffO: SQLite saves its database to a flat file.
    – Mr.Mindor
    Feb 6, 2013 at 21:07
  • 7
    @Mr.Mindor, most databases do... but thats irrelevant. 'Flat File' here means directly manipulating the file, as opposed to going through some DB layer. Feb 6, 2013 at 21:18
  • Disagree. I would still need to learn how SQLite work, and implement code that manipulate SQLite database in .NET, convert query result to objects, etc. so it doesn't make development easier. It just adds all the burdens of creating a database, without the advantages of a full-fledged database server.
    – Louis Rhys
    Feb 7, 2013 at 0:02

It's an example, used to demonstrate a concept, rather than a concept in and of itself.

The concept is that you don't make an architectural decision until the last responsible moment, but no later. But, in reality, you often do have a decision on the database you're going to use very early on. That might not be perfect, but it's a fact.

Once that decision is made, you don't actively avoid it. Storing something in an existing database is often as easy as storing it in a flat file.

But changing from MySql on Linux to SQL Server on Windows might not be as simple as changing from a flat file anywhere to SQL Server on Windows. That's the real point. So, while there's doubt as to the final solution, take the simplest possible option, accounting for change. Once a decision is made, stick to it.


What are you persisting?

I am on an agile team and for one application, we persist almost everything to the database. Mind you, if we didn't then there wouldn't be much for this application to do - persisting things to a database is a large part of its raison d'être.

So: What are you persisting and what does your application do? If the application doesn't actually care where its data is persisted, then you can write a small layer that makes the decision (that decision can be stored in a config file somewhere) of flat-file vs. database. I think you need to evaluate your application and your data and decide if it makes sense in your specific case to invest time in database persistence, or just read/write flat files (which will probably be faster in terms of development time).

  • 1
    The application manages a queue of requests, and it needs to remember the queue after closing and restarting.. there is no obligation to use database like in your application
    – Louis Rhys
    Feb 6, 2013 at 17:20
  • @LouisRhys: What will be done with this queue data? Simply reading and displaying to the user? What benefits do you think you will get from persisting in a database? Feb 6, 2013 at 17:21
  • the actions in the queue will be executed. benefit from database include performance, concurrency management, and probably the client will also care about security, readability & query of data.
    – Louis Rhys
    Feb 6, 2013 at 17:26
  • @LouisRhys: Maybe for the first iteration or two of development you could use a flat file, just to a get a proof-of-concept working, but you might want to completely separate the logic from the data access, since in the future it sounds like a database might be a good thing (and changing from file to db will take more time later). Ultimately, this is an advanced architecture decision that only you can make since you have access to the client's specs/requirements. Feb 6, 2013 at 18:05

A lot of people misconstrue that aspect of agile as meaning they shouldn't plan ahead for future features. Nothing could be further from the truth. What you don't do is allow planning for future features to delay delivering value to your customers now.

How that applies to something like persistence depends very much on your application. One of my current hobby projects is a calculator. Eventually, I would like to be able to store user defined constants and functions, and save the state when I close the program. That requires persistence, but I haven't even started to think about what form that would take. My program will be very usable without persistence, and adding it now will add significant delay to my first release. I would rather have a working calculator with fewer features than none at all while I wait for it to be perfect.

Another hobby project of mine is for video and photograph color correction. This application will be completely unusable without being able to save my work in progress, and the code needed to do so is pervasive throughout the entire program. If I don't get it right from the start, then changing it could be prohibitively difficult, so I've spent quite a bit of effort on persistence up front.

Most projects would fall somewhere between, but you should never feel bad about planning for future features if it adds little to no extra effort now. Databases may be complex, but most of that complexity is hidden behind a solid, well-tested interface. The work you will have to do in your application may very well be less for a database than a flat file, because of all the features a database gives you for free. There are "hybrid" options like SQLite if you don't want to deal with the hassle of a database server yet.

  • 1
    "Plans are useless, but planning is essential." - Eisenhower Feb 6, 2013 at 22:38

I think you are putting focus on the wrong values. In agile, business value is in focus. You create a product in order to deliver business value to some end users.

If you create the persistence layer late, or make it up along the way is your strategy for delivering business value to the customer. I don't believe that the term "agile" itself dictates if you should do one or the other.

The point of view about deferring data storage strategy is advocated in this presentation by Robert C. Martin (one of the authors of the agile manifesto).

It is a very good presentation, I can recommend that you watch it.

But I disagree with it! At least to a degree.

I do not believe that you can call a user story for "Done", if the user story involves data that should be persisted, and you don't actually have any type of persistence implemented.

If the product owner decides that now is the time to go live, you are unable to do that. And if you haven't started implementing persistence until late in the project, you also have no information about how long it would take to implement the persistence layer, leaving it a major project risk.

The agile projects I have worked on have not deferred the data access strategy. But it has been decoupled, allowing us to change it along the way. And the entire database schema is not designed up front. Tables and columns are created along the way as they are required in order to implement the user stored that, in the end, deliver business value.


It takes good judgement and experience to see what questions need answering first when embarking on a new project.

If the end product is still unknown then building/prototyping quickly well help you figure that out, and yes iterating in an agile way will help. That of course will introduce risk such as finding out late in the process that the persistence implementation is going to take longer than you communicated to your stakeholders.

If the end product is well known then understanding how persistence will work in your application could be more important to know early on. The risk there is that find problems with your product specification later in the development process.


Flat files are not simple!

They allow storage and that is all. The structure of the data, access paths etc. are all up to you, and, there are numerous ways to get this wrong.

There are reasons databases exist and one of them is to make things simpler for developers.

Most of my development is done for large systems within large companies. We always have a complete and well thought out data model before we procede with any further design or development. A data model helps you understand your problem and allows for a clean implementation.

Forgotten data items, mismatched data structures and other nightmares can all be avoided by producing a data model up front.

You can leave your actual choice of database technology till after the data model. But most "persistence" technologies are closely bound programming and even design styles. If you code for a relational database, and later decide to switch to a cloudy key value thingy you will need to completely re-write half your code.

If you start with flat files and switch to a relational DB you will probably end up throwing away half your code as your developers will have wasted their time implementing a piss poor database.


Should you try persistence in a flat file before database?

Yes, you should try it. Especially if you've never tried it before. No matter how it turns out, you'll learn something.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.