Software rewriting alternatives [duplicate]

Question

We have a legacy system to bring up-to-date because:

1. It uses an unpopular (amoung our users) non-sql database (Btrieve)
2. Provides only a text interface
3. Is written in Turbo Pascal (but compiled in free pascal) which is hard to recruit/retain developers to maintain it

To give you an idea of the scale, the code base is 15MB over 756 files.

All the studies I've looked in to such as those listed here suggest it is vastly more expensive to rewrite than to adapt what you have.

If I thought we could encapsulate the legacy stuff behind a web service and then design new client(s), that would be fine. But I'm stuck with this legacy database which 90% of those 756 files reference.

We even attempted replacing the low-level DB code with code to convert to SQL, and while it worked fine, it was too slow because the code is still accessing records one at a time generating far too many SQL statements than it would need. Plus this approach does not allow any improvements in the DB design.

When faced with such an upheaval as replacing the DB that is referenced so heavily in code and the other requirements listed, can it still pay to reuse or are we better off treating the new version as a whole new product and just listing the current systems abilities as requirements (bearing in mind no handy requirements doc exists for the current system).

I know it's too hard to give a definitive answer on this, so just any advice would be appreciated especially if you've experienced a similar situation.

Answer 1

Its true that re-writing is the most radical approach and should be avoided whenever possible.

Try to create an intermediate level between your application and the DB. It needn't be a separate project necessarily, but all DB requests should go through it. If you manage to do this first step, it will be easy to change the DB any time, you'll just need to re-write one single module of your project.

Once this is done, try to divide your project into small logical pieces and re-write ("translate") them one after another (if you really think it can't be avoided). If you have unit-tests, they would be a great help.

Answer 2

Before making any decisions about what approach that you would like to take, I always feel it is important for the development team to always consider the option of a complete rewrite as a possibility, even though many think it is nearly always a bad choice.

Take some time to write up some high level milestones that must be achieved, and make some very high level architectural decisions about what core technologies will comprise the new software. At this point make a rough estimate of the time you would feel is needed to complete such a project. It is good no matter what choice you make to at least have a good idea what a rewrite effort will cost you in terms of time and money.

The other very important cost of a rewrite that many neglect to consider can be the exorbitant cost of deployment in both money, time, and man hours. Users get comfortable with using older flawed systems, they memorize the quirks and bugs, the workarounds become part of their routine, they memorize the hotkeys to quickly perform their favorite tasks, they become surprisingly productive over time.

I have seen rewrites that by all accounts should have been a major success, the development came in under time and under budget, the major issues no longer exist, the newer system was easier and cheaper to find talent to maintain, the learning curve was smaller... the only major problem that had been neglected was user adoption and Agile based user involvement. When they were told that their text based mainframe system had been replaced with a GUI based desktop client, they were unfamiliar with it and productivity dropped. User input only came from a single individual who no longer performed these job functions so important key combinations, lack of keyboard support, hot keys, poor tab ordering, and the need for a lot of point and click mouse interaction caused productivity to plummet. Sure they would have gotten comfortable and productive eventually, and without the major bugs and improved performance they would have been vastly more productive with the new GUI based system once the kinks were worked out.

What unfortunately happened is that there was open revolt against the new system, and despite management buy-in, they could not force the users on the ground to adopt it. The project failed at deployment. The hidden costs and risks of deploying and training for a rewrite are important to consider when comparing to the increasing cost of maintaining the legacy system.

Another option to consider is to look into the technologies that are causing the most grief, most expensive to license, hardest to find talent for, inherently buggy, etc. A good start might be to rewrite small pieces of the application in such a way as to decouple the core components. Once this is done it is easier to swap and replace components as time becomes available to rewrite them, and this greatly reduces the risk of refactoring in general.

If I were you I would start with replacing just the database. You can use one that is more popular and that is more easily maintainable by the talent pool. As you replace the schema you can slowly start seperating database access specific logic into seperate modules, making it easier to decouple the business logic from the database implementation and database access code. It can be justified in improved performance, decreased cost of maintainability, and quicker turnaround.

Answer 3

Some random thoughts:

Big-bang rewrites are prone to big-bang problems, which is why many people suggest an incremental approach to rewriting an application. Ideally, then, you need to Divide and conquer.

If you're going for an incremental approach, the challenges are:

• How you divide the existing application into upgradable components
• How you upgrade each component
• How you integrate new components with the old ones

Look for logical divisions that already exist in the application. Are there separate pieces of hardware? Different technologies? Layers? Does it serve different classes of business user? Or perform different groups of business function?

The better organised your original code, the easier your task will be. As a result, breaking up an application in this way will almost certainly involve some amount of refactoring of the original application. For example, pulling out database calls into a single set of modules / classes. Be aware that you're not trying to improve your old application - your purpose is just to modularise it. Fixing other problems should be easier in the new technology (after all, that is presumably one of the reasons you're migrating in the first place).

The smaller the chunks you're dealing with the easier your task will be. So take things a step at a time. Even if you only manage a few hundred lines of code, at least that is a start.

If you can automate the upgrade process in any way, then do so. For example, you may be able to buy / make a tool to convert program code or database schemas.

Write unit tests for both old and new code.

Answer 4

Your suggested approach - encapsulate the legacy stuff behind a webservice is a good practice to follow. it will ensure that business logic will stay same as it was, and minimize the development time for better looking application.

Consider some No-Sql databases currently available in the market.

Currently, we are using Raven DB for the project needs. Although, it is quit odd to start getting into it. However, it seems quit a good match for a pure DDD approach.

Here is a post explaining strong sides of Raven DB among the No-Sql databases - Why Raven DB?

Answer 5

Two issues.

One, the tiger team issue. Usually these developments fail as the old system would be the requirement for the new one, while the old one also changes, it's a Zeno's paradox. Uncle Bob wrote an article about this, but it's also part of several of his talks and perhaps some of his books as well.

Second, the UI issue.

It's not just that people are used to the UI. That's the tip of the iceberg.

For the users, the system IS the UI. That's why it was called the "user interface" at the first place. I don't even really understand how could they hate the underlying database system, as they have nothing to deal with it? Isn't it lying "under" enough?

If the user interface, the behaviour, the data, and the performance stays the same, you could change every bit of the system, the users wouldn't take a notice. You could convert it from Borland Pascal to Turbo C and vice-versa, you could change the server OS from FreeBSD to Solaris, from Windows to Linux, from Intel to PPC, whatever.

And this is good and bad for you. As long, as you keep the UI, you can change the system. If you change the interface, make sure the logic stays the same behind for a while.

(Your only problem will be, that the users don't have a problem with the underlying system: they have a problem with its user interface. Every single user-initiated change request is UI-based, this is what SVN / CVS / git logs show of every system, because that's the language the user speaks.)

It's like refactoring: you either change the code structure, but don't modify behaviour, or you change the behaviour, but don't change code structure. And have automated tests for that.

I don't know how you can test it, usually we used screenscrapers for reaaallly old applications. I've even seen banking frontend which literally typed the results of a form submission into a terminal to the old code. Not joking.

Change one step at a time. Have a hybrid system for a while. Change UI, retain system, change system, retain UI, change UI, retain system, change system, retain UI...

It's good that you have the source. Even if it's Pascal, it makes your life a few billion magnitudes easier.

For one refactor, I collected all input the system got in a half-an hour intensive testing period, and collected all the database changes it was to do. You can read about it here

Perhaps you should collect real system usage, by the use of keyloggers and terminal capture like kibitz or script.

Oh, and pray that your tests cover everything.