11

Recently I had a discussion with a developer who mentioned that during program development, they routinely create and delete tables and columns on a regular basis while working on new features and justified things by saying that this is normal when using an agile development process.

As most of my background is in a waterfall development environment, I wonder if this is actually considered proper under agile development, or if this might be a sign of an underlying problem, either with the program architecture or with their following of the agile process.

Improve this question
12
  • One comment on the database, the last time it was checked there are over 700 table and I've heard mention by others that there "isn't enough time to refactor it."
    – rjzii
    Feb 29, 2012 at 16:07
  • 24
    700 tables and not enough time to refactor? I can smell the fail all the way over here.
    – Adam Crossland
    Feb 29, 2012 at 16:10
  • 7
    700 USED tables or 700 tables many of which are orphans?
    – Ben Brocka
    Feb 29, 2012 at 16:17
  • 1
    @AdamCrossland Seriously... Lynyrd Skynyrd's Ooh That Smell comes to mind. But on a serious note, denormalized tables sometimes are a good design choice on read-heavy or databases that have a heavy reporting load.
    – maple_shaft ♦
    Feb 29, 2012 at 16:23
  • 1
    @maple_shaft Sure, any technology can be abused, like anything else you have to weigh the pros/cons and test, test, test. Materialized views may, however, offer you an out if you find yourself denormalizing your tables. I think your point is just further reason to have a DBA or at least a developer with strong DB-fu on staff to pull the reins back when everybody else is charging ahead with a clearly poor design. My best work has come not while coding or designing, but preventing others from making horrible, horrible decisions.
    – Mike Cellini
    Feb 29, 2012 at 18:21

11 Answers 11

Reset to default
21

It's becoming more-and-more apparent to me every day that "agile" is becoming a synonym for poorly thought-out, chaotic, hurried and seat-of-your-pants. And none of those things are compatible with an Agile approach as I understand it.

Having an effective and repeatable Agile process is not easy, and I don't believe that it inherently reduces the total amount of work to be done even though it may very well lead to better products.

If they've said that they don't have time to "refactor" the database then they probably also don't have time to set up versioning and migration for the database. They probably haven't taken the time to create a suite of functional tests for it. All of those things are what I think of when I think of a solid Agile process that's headed for success.

In the end, Agile is just a word. What you are doing day-to-day determines if you'll be successful or not.

Improve this answer
4
  • 2
    Having effective Agile process should actually mean more upfront work because of the repeated focus on delivering only functional software after every sprint. If done right then it leads to better chance of success. Waterfall is actually much more efficient if you assume that requirements are static and project resources don't make mistakes. This is fantasy in most situations though.
    – maple_shaft ♦
    Feb 29, 2012 at 16:29
  • 1
    @maple_shaft, just so. Being Agile doesn't remove the hard work from building products.
    – Adam Crossland
    Feb 29, 2012 at 16:31
  • 2
    I have a feeling if this team were using a waterfall approach, it would be just as chaotic and any change request would be a disaster.
    – JeffO
    Mar 1, 2012 at 0:50
  • Well said, Jeff O.
    – Adam Crossland
    Mar 1, 2012 at 1:30
11

While it is not unusual to create and drop tables as a design evolves, some cleanup might be in order to make sure your database is really using all those tables.

Yes, Agile is all about refactoring, but if they are now saying the system is too big to refactor, they have stopped doing Agile and are now just Cowboy programming. The development team won't like being called that, but that is what they are doing. Riding the range shooting anything that moves.

A DBA will help, just make sure you get a DBA that understands development as well as Agile development. Your development team needs to be reined in, not thrown in jail.

Improve this answer
5

In general, often creating new tables and adding new columns is very normal in a process where programming and database administration are not strictly separated. A new feature might create new data that must go somewhere. Try too strictly to avoid that and you end up with an inner-platform model.

Well-written software hardly notices those new database objects, so nothing breaks just because of a new column in some table.

On the other hand, regularily droping columns or even tables is suspicious. A new feature never needs a table removed, so this could be a sign of people working completely planless.

Improve this answer
3
  • 1
    The creation of new tables and columns doesn't bother me when it is justified, but it has been pointed out that removing tables (and columns for that matter) generally means that some work was lost due to someone planning inappropriately or someone else deciding that the feature was not needed after all. Likewise, the shear number of tables is a concern due to the lack of normalization in them.
    – rjzii
    Feb 29, 2012 at 17:04
  • Exactly. Don't worry about the large number of tables, most of them are unused anyway and maybe they'll be dropped soon. SCNR
    – user281377
    Feb 29, 2012 at 17:37
  • Well, the problem is that they are all be used, even if it is only for a single record.
    – rjzii
    Feb 29, 2012 at 17:48
4

If your database can be easily versioned and migrated and you've got the tests to prove that changing things did not break things then you've probably got a pretty agile process going.

In light of the comments -- generally to the effect of these are a bunch of cowboys justifying themselves as agile -- run screaming. Fast. And post all that you can to thedailywtf.com so we can all enjoy your horror.

Improve this answer
7
  • The sad thing is that the database isn't easily versioned, migrated, and there is limited testing in place at best. Developers frequently overwrite the changes made by other developers.
    – rjzii
    Feb 29, 2012 at 16:18
  • 5
    Definitely Cowboy programming then. If you have a management team behind this "Agile" effort, make sure to rat out this team to them that they are abusing Agile and just running amok.
    – Bill Leeper
    Feb 29, 2012 at 16:19
  • 1
    @RobZ Agile is not an excuse for poor unit test coverage and poor database design. That sounds like a hot mess.
    – maple_shaft ♦
    Feb 29, 2012 at 16:21
  • 2
    ugh! not versioned!!! no wonder it's a mess. I shudder to think what the app code is like.
    – ozz
    Feb 29, 2012 at 16:22
  • 4
    Basically this team is not doing anything agile, they are just an AdHoc team. If there is a management structure in place, you could anonymously or in person raise concerns up the chain. Tell them you are seeing a huge disaster in the database, lack of tests, and improper practices being used to manage code. This team is headed for a huge FAIL.
    – Bill Leeper
    Feb 29, 2012 at 16:25
3

Like most answers here on StackExchange, the reply should be 'it depends'. In agile development, requirements and specifications are discovered during implementation.

However, given agile development, when a relational model is properly normalized, adding new attributes to relations should rarely be a necessity , new entities should generally refer to older ones, given a proper model.

Most developers don't normalize their DB's because of time-constraints, laziness, incompetence or query complexity. Renormalizing requires transfer of existing data, modification of the DAO's, etc. etc. which generates a risk-factor.

Improve this answer
3
  • At what point in the agile process does the proper model for all future change requests magically appear?
    – JeffO
    Mar 1, 2012 at 1:01
  • After properly studying the domain. There is a limited number of properties which completely define an entity. Some (increasingly complex) examples: a 2D point is completely defined by two coordinates, an address is completely defined by a country, a fields version and a realization of a set of address fields defined by another entity (using a country id, a version and constraints).
    – Dibbeke
    Mar 1, 2012 at 11:17
  • @Dibbeke: And then you get business issues like treating the EU (27 countries) as a single country in cases X, Y and Z. Or treating US states as countries. Or the business realization that some DB entries have 2 address representations for a single address.
    – MSalters
    Mar 1, 2012 at 14:21
3

To answer your question, no, that isn't normal in an Agile process.

Where it may seem to stem from an Agile attitude is from Agile's short-iteration design/develop/test cycle, and Agile's emphasis on lightweight solutions that meet the known requirements, but are well-structured in order to be able to meet new requirements with minimal change. Given these two things, you might say that a developer, not knowing what might come down the line but knowing his change shouldn't impact the DB in a way that can't be undone, simply makes the necessary changes to the schema in the "lightest" way possible, and then at intervals several sets of "light" changes will be refactored into something more permanent and stable.

Having said that, I have not yet worked with a developer who subscribed to Agile theory and methodology, and also thought that routinely creating and deleting tables in the schema was necessary for any reason. Agile doesn't mean slap-dash or bolt-on. If you are given a story that requires the addition of a new field of data belonging to a particular record, you add the field to the table. If that change breaks something, you figure out why, and make other changes as may be necessary (I can think of very few things that would break by adding a column to a DB being queried; if it does break with this kind of change you have bigger problems). Refactoring is normally limited to code; changing the schema is usually a more involved process than changing code, and so when schema changes must occur they are usually made with more care, and at least some attention paid to knowledge of the future direction of the project. Having to re-structure some or all of the database indicates a fundamental failure of design; being Agile doesn't mean there isn't a "master plan" of basic architecture and design rules to follow while organically building the program and data structure.

Now, there are cases in Agile where what you "know" now will be contradicted by what you will learn later. Say you have a requirement that your system must have an Address for every Person. Since this is a 1:1 relationship and the data will be needed in the majority of cases, you simply add the Address fields to the Person table. A week later, you receive a requirement that a Person can have more than one Address. Now it's a 1:N relationship, and to properly model it you must undo your previous changes, splitting the fields out into a new Address table. This is not routine, especially among senior developers. An experienced developer will see that a Person has an Address, consider these as conceptually separate, and create a Person table and an Address table, linking Person to Address with a FK reference to an AddressID. A schema such as this is easier to change should the nature of the relationship change; without creating or deleting entire "wide" data tables, the relationship between Person and Address can be pretty easily modified from 1:1 to 1:N to N:N.

Improve this answer
2

There isn't as much focus on design up front when working under Agile, so I don't see this as being a huge problem, certainly for the first release.

It is hard to comment on a system that has 700 tables I haven't seen, it may well need all of them, but it could also be the case that the database hasn't been managed well enough.

Even under agile, for a big system, it is quite common to still have someone/team in charge of the schema.

Improve this answer
2

If they are making such changes frequently - particularly dropping tables and columns in live applications - that seems like a sign of inexperience. It has nothing to do with whatever process they are claiming to follow. 'Agile' is not an excuse to not sit down and think about the data you need to store and how it relates before you start pounding out code. If they find they are altering more than 10-20% of the initial structure, to me thats an indicator they're either not thinking things through or they dont have a lot of experience analyzing requirements and designing databases, and so they simply get too much of it wrong at the beginning.

Improve this answer
2

While it sounds like your team is indeed simply cowboy coding, there really should be nothing wrong with refactoring code OR databases. It's not lost work--it's adapting to a newly learned reality.

I'd suggest what the team needs is a sandbox to try out changes, do some testing, bounce them off the users, and decide whether they make sense. At that point, integrating the changes that make sense--with adequate regression testing--into your schema should be fine.

Improve this answer
1

Agile is about coding, Databases aren't code. Changing a database should be treated like remodeling a house. People somehow got the belief that agile means act now plan later, which is completely untrue. Even under agile methods time needs to be given for planning, especially for something as important as database design.

Improve this answer
4
  • 5
    Databases are not code, but schema is and should be treated as such. It should be versioned and source controlled too. I want to downvote this but I agree too much with the rest of your answer though.
    – maple_shaft ♦
    Feb 29, 2012 at 16:50
  • 6
    Agile is not about coding. It's about the software development process, which most definitely includes databases if the working software depends on the database. Having said that, I agree with your assertion that Agile does not mean 'act now plan later'.
    – Eric King
    Feb 29, 2012 at 16:54
  • @maple_shaft just because a db schema is treated like code doesn't make it code. pets are treated as people, but it doesn't make them people
    – Ryathal
    Feb 29, 2012 at 20:55
  • 3
    Whether something is code or not, it needs to be planned. Changing a database should be treated like changing code along with considerations for the existing data. It actually takes more thought and planning.
    – JeffO
    Mar 1, 2012 at 0:56
1

My last job was on a team like this. When using an agile process requirements change. Sometimes the changes mean a existing entity needs a new attribute resulting in a new column in an existing table or a whole new entity is required resulting a new table with relationships to existing tables. These kinds of changes come with the territory and can't be ignore just because you don't want to touch the schema. Success is determined by maintaining the integrity of your data when migrating from one database version to the next and proper unit testing.

Just try not to make unnecessary changes to your schema. For example, if a feature requires a new table to be created, make sure you're happy with the definition of that table before you check it in and roll it out to your test environment. Having to undo a change to your database schema after your data has been migrated can be painful.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.