The cycle you describe is normal. The way to improve things is not to avoid this cycle, but to streamline it. The first step is to accept that:
- It's near impossible to know everything on day one of a project.
- Even if you do somehow know everything, by the time you've finished the project then something (the client's requirements, the market they're in, the tech you're working with, their customers' wishes) will have changed and made at least part of what you knew invalid or incorrect.
Therefore, it's impossible to plan everything up front, and even if you could, following that plan would lead you to build something imperfect or obsolete. Knowing this, we integrate change into our planning. Let's look at your steps:
- Start with a few use-cases
- Start coding
- Realize a few things I did not handle well, and does not fit well in current codebase.
- Rewrite most part of code
That's actually a great starting point. Here's how I'd approach it:
1. Start with a few use-cases
Good. By saying "use cases", you're focusing on what the software is for. By saying "a few", you're not trying to discover everything; you're sticking to a manageable amount of work. All I'd add here is to prioritise them. With your client or end user, work out the answer to this question:
What is the smallest, simplest piece of software I could give you that would improve your situation?
This is your minimum viable product - anything smaller than this isn't helpful to your user, but anything bigger risks planning too much too soon. Get enough information to build this, then move on. Be mindful that you won't know everything at this point.
2. Start coding.
Great. You get working as soon as possible. Until you've written code, your clients have received zero benefit. The more time you spend planning, the longer the client has spent waiting with no payback.
Here, I'd add a reminder to write good code. Remember and follow the SOLID Principles, write decent unit tests around anything fragile or complex, make notes on anything you're likely to forget or that might cause problems later. You want to be structuring your code so that change won't cause problems. To do this, every time you make a decision to build something this way instead of that way, you structure your code so that as little code as possible is affected by that decision. In general, a good way to do this is to separate your code:
- use simple, discrete components (depending on your language and situation, this component might be a function, a class, an assembly, a module, a service, etc. You might also have a large component that is built out of smaller ones, like a class with lots of functions, or an assembly with lots of classes.)
- each component does one job, or jobs relating to one thing
- changes to the way one component does its internal workings should not cause other components to have to change
- components should be given things they use or depend on, rather than fetching or creating them
- components should give information to other components and ask them to do work, rather than fetching information and doing the work themselves
- components should not access, use, or depend upon the inner workings of other components - only use their publicly-accessible functions
By doing this, you're isolating the effects of a change so that in most cases, you can fix a problem in one place, and the rest of your code doesn't notice.
3. Encounter issues or shortcomings in the design.
This will happen. It is unavoidable. Accept this. When you hit one of these problems, decide what sort of problem it is.
Some problems are issues in your code or design that make it hard to do what the software should do. For these problems, you need to go back and alter your design to fix the problem.
Some problems are caused by not having enough information, or by having something that you didn't think of before. For these problems, you need to go back to your user or client, and ask them how they'd like to address the issue. When you have the answer, you then go and update your design to handle it.
In both cases, you should be paying attention to what parts of your code had to change, and as you write more code, you should be thinking about which parts may have to change in the future. This makes it easier to work out what parts might be too interlinked, and what parts might need to be more isolated.
4. Rewrite part of the code
Once you've identified how you need to change the code, you can go and make the change. If you've structured your code well, then this will usually involve changing only one component, but in some cases it might involve adding some components as well. If you find that you're having to change a lot of things in a lot of places, then think about why that is. Could you add a component that keeps all of this code inside itself, and then have all these places just use that component? If you can, do so, and next time you have to change this feature you'll be able to do it in one place.
A common cause of issues in software is not knowing the requirements well enough. This is often not the developers' fault - often, the user isn't sure what they need either. The easiest way to solve this is to reverse the question. Instead of asking "what do you need the software to do?", each time you go through these steps, give the user what you've built so far and ask them "I built this - does it do what you need?". If they say yes, then you've built something that solves their problem, and you can stop working! If they say no, then they'll be able to tell you in more specific terms what's wrong with your software, and you can go improve that specific thing and come back for more feedback.
As you go through this cycle, pay attention to the problems you're finding and the changes you're making. Are there patterns? Can you improve?
- If you keep finding you've overlooked a certain user's viewpoint, could you get that user to be more involved in the design phase?
- If you keep having to change things to be compatible with a technology, could you build something to interface between your code and that technology so you only have to change the interface?
- If the user keeps changing their mind about words, colours, pictures or other things in the UI, could you build a component that provides to the rest of the application those so that they're all in one place?
- If you find that a lot of your changes are in the same component, are you sure that component is sticking to just one job? Could you divide it into a few smaller pieces? Can you change this component without having to touch any others?
What you're moving towards here is a style of working known as Agile. Agile isn't a methodology, it's a family of methodologies incorporating a whole load of things (Scrum, XP, Kanban, to name a few) but the thing they all have in common is the idea that things change, and as software developers we should plan to adapt to changes rather than avoiding or ignoring them. Some of its core principles - in particular, the ones that are relevant to your situation - are the following:
- Don't plan further ahead than you can predict with confidence
- Make allowances for things to change as you go
- Rather than building something big in one go, build something small and then incrementally improve it
- Keep the end user involved in the process, and get prompt, regular feedback
- Examine your own work and progress, and learn from your mistakes