How can we effectively manage software projects without killing creativity?

Question

I am convinced that software development is essentially a creative process. I also believe that this is the case for all levels, from architecture to coding.

What makes me think so? To put it very briefly, because a software developer is supposed to create something new, not just copy existing stuff. It is more than just grabbing into your toolbox and getting out the right tool for a job, although it definitely helps to have a good toolbox.

On the other hand, when we view software development from the project management side, it is desirable to split a development project into small tasks and assign to each task a certain time in which it is expected to be completed. (I know that there is the concept of story points, but I don't think it makes such a big difference in practice. At the end of the day, a developer is expected to deliver after a certain amount of time.)

From a project management perspective it is clear that these tasks should be small. Depending on whom you ask, an ideal task should be something between 30 minutes and a day.

Now here's my problem: I find it hard to be creative when facing a time limit for a job, even if it is a soft limit based on estimates or story points. The shorter the time limit, the worse it gets. Often I feel that I am much more productive (because I have the freedom to be creative) when I just do what I think needs to be done without thinking too much about the planned times for each task. Some tasks might take much longer than expected but the quality will be higher and on the whole, the project will probably be finished earlier.

Is this just my personal perception or is it a general problem. If the latter, what can be done about it?

Edit
After reading the first comments and answers, I remembered (once again) that the term creativity does not have the best reputation in engineering. By being creative I do not mean doing uneccessary stuff that provides no business value. What I mean is solving problems in new or non-standard ways.

Answer 1

You are mistaken what "manage" means. In context of software development, managing a project means that the project progresses with necessary features, on budget, on time and with acceptable quality.

Splitting the project into bite-sized chunks is one of the many ways how it should be possible to achieve above goals. And as you imagine, it might not be optimal way for you. Personally, that approach insults me, as it treats me, a developer, as irresponsible and unable to properly progress on the project, unless micro managed.

If you really want to properly manage a developer, you should treat him as a responsible, skilled person who knows what he is doing and is willing and motivated to do anything necessary for a project to succeed (within reason, of course). Your primary management responsibility, would be to clearly communicate goals of your project and leave the day-to-day management to the developers themselves. You should track progress not based on how many tasks were completed, but on how good the software matches your goals.

And if the developers themselves think splitting the project into small tasks and tracking those tasks is beneficial for their work, they will do just that. But this should be completely unrelated to what you, as a manger, do.

Answer 2

I would advise you to think of your software development in the context of the business. The business' goal is to make a profit. To make a profit the business needs to produce as fast as it can while maintaining a level of quality.

When we're deep in the designs and the code, it's sometimes easy to forget that often the simplest, straightforward, and fastest method is best for the business, but that's not always the case. Sometimes it takes a lot of consideration and creativity to complete a feature.

My suggestion is you take this all into account when estimating. If this is an important feature that spending more time on will dramatically improve the quality or improve the agility of the rest of the project, then estimate higher. If you're consistently struggling to produce a decent result in the time you've estimated, then estimate higher. Propose quality tradeoffs to the project manager and get their opinion. If early into working on the task, you think more time could produce a better result, consult your project manager to get his opinion about whether a better result is worth the estimate slip. In this way you share the responsibility of the schedule and the quality with the rest of your team.

In a perfect world we wouldn't need to put estimates on anything and we would have unlimited time to create beautiful and elegant software, but mostly that's not the case, and we must accept that business needs must be placed foremost.

Answer 3

Estimates are not deadlines.

An estimate specifies how long such a task takes on average, but particulars of that task may cause it to be easier or harder that expected.

If you treat estimates as deadlines, either the deadlines will often be missed, or estimates will be padded to something like the 90th percentile. This makes estimates useless for estimating total remaining effort (you may recall from statistics that the average of the sum is the sum of the averages, but the 90th percentile of the sum is lower than the sum of 90th percentiles. In fact, for sufficiently many tasks the strong law of large numbers implies that the 90% percentile of the sum gets close to the sum of averages, so if you want to add up estimates, you need averages, not guarantees).

Project management being concerned with managing the project, having a good estimate for project effort is more important than good task estimates, so a good project manager should allow for considerable variation for task completion times. Over the course of weeks or months though, the averages should come close to the estimate.

Answer 4

From my personal view boring projects kill creativity, not software management in general.

An interesting problem (project) typically requires you to perform at your best and utilize what you so far have learned. If you find that you're coming up simple barebone solutions which are boring it's very likely because the project is of that nature.

Imagine that I ask you to create the same web page four times over with only very slight design differences and say that you can't reuse the codebase because of legal reasons. Your creativity is much more likely to be stifled by the repetition of the task rather than any time schedule set.

I have also done maintenance work on projects which could be considered in cryostasis, which is not a very attractive prospect either. It fits the approach of dust off the code, fix it and run away fast and you should do so, because you won't ever get to nurture these products to health.

Your view obviously stems from your own perception of the reality you've worked in but as I've portrayed there are cases in which I find that it holds true; typically when management does not appreciate the value of spending more than the minimum necessary time on any solution or when the solution is pre-determined. The latter of those two points deserves a small paragraph.

I am reminded of some passages in The Mythical Man-Month which discusses the nature of creativity in software. It implied heavily that creativity is possible at low implementation level so long as the implementation is free for the implementer to design. Essentially, if I tell you to write a function in a specific way there is not much creativity involved as opposed to if you would be to write it from scratch on your own though with a predefined input and output.

Time to research new techniques, patterns and technologies is something I value highly. Quite often I can be found researching something unrelated to my task during my working hours. What's important here is that it has proven to provide a lot of value in the long run. We're talking security fixes due to discovered vulnerabilities, design patterns, optimizations to no longer performant code and so on. It's important for there to be a balance though, you obviously can't drain all your time like this.

So yes, I think there are things that stifle creativity. What can we do about them? Frankly most of the things are dependent on the project and/or management, you can highlight your opinions and hope for change or look to take your services elsewhere.

Most importantly though: Accept the non-creative solutions when that's what is best and spend your energy on problems which require it.

Answer 5

From reading the OP, I see 2 questions here,

1. Creativity: I don't see a separation between tasks & creativity. The dev team should be setting how to complete the task and fulfill the requirements. With that, the dev lead can define task goals and timelines that give developers the freedom to be creative in their solutions, within the limits of meeting the requirements and meeting corporate coding standards/policy.

Yes, project managers and stakeholders will drive towards deadlines, that's part of their responsibility. But if the dev lead has done their job, then there is room for creativity. And frankly, in the end, there will be times when you just have to get the thing done the quickest way you can. Part of the developer world.

2. Tasks: while the nature of projects requires tasks to have a time estimate, I've seen serious issues caused when tasks are broken down too finely with an arbitrary limit (like 8 hours). That can create a monster list of tasks that becomes unmanageable and unreportable. Better to create tasks that represent a logical unit of work. For example, "code new customer entry screen". The nature of development is that you will have iterations, you will have sticky points that require more time. Trying to define tasks to the level of "write code for customer entry screen required fields", "unit test required fields" and link them in a sequential fashion will be illogical, due to the code - test - debug - fix code - test..... cycles. Project managers need to think of logical work units, not conforming to some arbitrary model of A Task.

Answer 6

If you're concerned about time constraints, I typically explain the overall situation to project managers in the following kind of way.

I always work as fast as I can, but not faster. Too fast just leads to additional mistakes/bugs that take longer to fix than simply working more carefully in the first place. So working at my maximum >>effective<< speed is what will get the job done in the shortest possible calendar time. And if that's not fast enough for your schedule, then either the schedule's unrealistic or you've got the wrong man (me) for the job.

Please note that (as you can imagine) I always word that last sentence lots more "gently". I'm just telegraphing the idea to you here. Nevertheless, you've somehow got to get that idea across.

Edit (re your edit emphasizing "business value")
Sometimes tight schedules are dictated by "physical/social reality" rather than by the capriciousness of project managers. And in those kinds of cases "creativity" can mean finding some way to make a long job shorter. Here are two examples from my own experience (see my resume, homepage url on profile, for details, including code and documentation samples).

Back (way back) in 1986, I had a contract with CBS as lead programmer to help develop their "Electronic Display System" for the midterm elections that year. So the deadline was Nov 4, period!

Also way back, in 1989, I had a contract with Chase bank to develop a fixed price trading system for government (and later corporate) bonds. While there was a bit more wiggle room in this schedule than CBS's, the system also incorporated various SEC regulatory changes, and Chase would have potentially incurred some significant penalties had the schedule not been met.

Several more recent projects had similar schedule rigidity (but I don't discuss clients after y2k online). In any event, all these kinds of situations require substantial creativity directed towards getting the job done correctly and on schedule. There are all kinds of "creativity", and your toolkit should include these kinds, too.

Answer 7

Now here's my problem: I find it hard to be creative when facing a time limit for a job, even if it is a soft limit based on estimates or story points.

I actually find the opposite, in a way. In my experience, a time limit can be a creative constraint that forces you to question the usefulness of each initial requirement. When you're short on time, you also have to develop workarounds and find new, cheaper or smarter ways of approaching a problem.

At the end of the day, you might have traded off quality, extensibility or scope for time but can you really say you were being less creative?

In other words, does being creative necessarily mean to create a lot of things?

Edit about technical creativity : it shouldn't be something you totally improvise by yourself at the last moment

In my team, much of the "creativity" in code design happens as the devs are collectively breaking down a user story into tasks and estimating them. This is when we'll have technical discussions about the nitty gritty design details, do our white board brainstorming and schema drawing. The idea is that you don't come up with a task estimate at random. It's something everybody extensively thought out and agreed upon.

Another big part of the creativity happens in spikes, technical research and exploration tasks that also have a time limit. Again, the creative constraint of time plays an important role here. It forces you to focus on coming up with a few pragmatic solutions and alleviates analysis paralysis.

All of this narrows down the amount of creative freedom a single developer needs when doing a task. Of course, they still have the liberty of naming things the way they want to name them, designing their unit tests as they see fit, etc. but the broad picture should have been set collectively before.