Advice on designing web application with a 40+ year lifetime

Question

Scenario

Currently, I am apart of a health care project whose main requirement is to capture data with unknown attributes using user generated forms by health care providers. The second requirement is that data integrity is key and that the application will be used for 40+ years. We are currently migrating the client's data from the past 40 years from various sources (Paper, Excel, Access, etc...) to the database. Future requirements are:

• Workflow management of forms
• Schedule management of forms
• Security/Role based management
• Reporting engine
• Mobile/Tablet support

Situation

Only 6 months in, the current (contracted) architect/senior programmer has taken the "fast" approach and has designed a poor system. The database is not normalized, the code is coupled, the tiers have no dedicated purpose and data is starting to go missing since he has designed some beans to perform "deletes" on the database. The code base is extremely bloated and there are jobs just to synchronize data since the database is not normalized. His approach has been to rely on backup jobs to restore missing data and doesn't seem to believe in re-factoring.

Having presented my findings to the PM, the architect will be removed when his contract ends. I have been given the task to re-architect this application. My team consists of me and one junior programmer. We have no other resources. We have been granted a 6-month requirement freeze in which we can focus on re-building this system.

I suggested using a CMS system like Drupal, but for policy reasons at the client's organization, the system must be built from scratch.

This is the first time that I will be designing a system with a 40+ lifespan. I have only worked on projects with 3-5 year lifespans, so this situation is very new, yet exciting.

Questions

• What design considerations will make the system more "future proof"?
• What questions should be asked to the client/PM to make the system more "future proof"?

Answer 1

Data is King

I think its a bit unreasonable to expect a web application circa 2013 to be still up and runnable in 2053. Technologies are going to change. Platforms are going to come and go. HTML may be a quaint memory by then. But your data will still be around.

So data is your primary focus. As long as your data is still there, people will be able to adapt to whatever new technologies come about. Make sure your data schemes are well thought out, and well suitable for expansion. Take your time spec'ing them out.

Regarding the actual applications, your company is probably correct here in having a 'build from scratch' directive. I maintain a couple 10+ year old web apps, and I'm very glad they are not locked into the prevailing CMS systems of 2003. They use home grown, very simple frameworks. I think for something like this you are better off with a very basic framework that you create specficially for the needs of the project.

But the reality is, over 40 years, the company will (hopefully) be making quite a few front-end and back end services to adapt to evolving platforms. So given that, I'd target a 5-10 year lifetime for individual user-facing applications.

Answer 2

We produce software that has been in use by paying customers for over 20 years. The codebase has outlasted several generations of source control tools. Our software hits all your bullet points except for the tablet thing.

Some of the concerns include ESIGN and UETA. Our lawyers believe that we need to keep electronic records readable for a minimum of 10 years. For documents that are retained whole, you should look into PDF/A.

For your database, don't worry too much about normalization. Instead you should be concerned about logging everything and having audit tables that track changes/deletes in data. When upgrading versions, plan on testing new versions in parallel for enough time to ensure that you've got your data migrated. This testing of new versions also includes new operating systems - we've had some very unpleasant surprises over the years. Preserve installation media and license keys in the event that a rollback needs doing. Test backups. If you are going to serialize objects to store in the database, do so as XML instead of the serialization supplied by your development framework.

For your staffing, long term code bases need long term memory. Ideally you'd want people around who have been around a long time. If that is institutionally impossible, then you need to document everything in something like a wiki. And my advice is a wiki that can tie in with your bug tracking system.

For your codebase, make sure you have comments in your code. Migrating from one version control system to another will almost always lose your check-in comments. I'm a fan of naming unit tests after spec and bug numbers. That way if the unit test Test_Bug_1235 breaks, then you know what and where to track down what it is supposed to be testing. It isn't as "sexy" as naming your tests Check_File_Save_Networked_Drives but that sort of test is hard to backtrack to specifications, requirements or bugs unlike Test_requirement_54321_case_2.

Answer 3

Rather than trying to figure out how this application is going to still be in operation 20 years from now, I think you should spend your six months fixing the problems you found that the original architect caused, put in place a sensible, robust architecture, and move forward from there.

Partial de-normalization of a database is not necessarily entirely unexpected in a medical setting. Some parts of medical databases have characteristics which make them a good fit for the EAV (Entity/Attribute/Value) model.

Answer 4

Answer from a front-end perspective:

Don't listen to everyone saying it can't be done, because an experimental San Francisco State University web service I co-wrote in 1996 finally went to Internet heaven a couple years ago, and never needed a single browser compatibility fix in that time; that's almost half of your 40-year goal. And this JavaScript-based front-end I made in 1998 for a Stanford Research Institute project was replaced with something flashier a few years later, but there's no reason the original UI couldn't still be running today with minor compatibility fixes.

The trick is to ensure your app uses only widely-supported W3C/ECMA standards and has a clean design under your control. While a lot of web apps written to the trendy 90s-era technology won't work well or at all today, 90s-era web apps written to major standards still do. They may look passé, but they work.

The goal here isn't to write a web app that will go up on their server and remain there for 40 years without anyone touching it again. It's to build a foundation that can still be in use decades down the line, which can grow to support new features without having to be rebuilt from scratch.

First of all, you have to code to official standards and only to official standards. No JavaScript features not part of a ratified ECMAScript standard; ES5.1 is the current version and is generally supported so that's safe to target. Likewise, current versions of HTML5, CSS, and Unicode are good. No experimental JavaScript, CSS3 or HTML features (the ones with vendor-prefixes or without 100% agreement between browsers). And no browser-specific compatibility hacks. You can start using a new feature once it's in the standard and everyone supports it without prefixes.

ES5 support would mean dropping IE8 or earlier, which I suggest anyway since it requires browser-specific hacks which will be useless in a couple of years. I'd suggest ES5's Strict Mode for the best chance at longevity, which actually sets your baseline browser compatibility at IE10 and recent versions of everyone else. Those browsers also have native support for many of HTML5's form validation and placeholder features, which will be useful for a very long time.

New editions of ECMAScript maintain compatibility with older versions, so it'll be much easier to adopt upcoming features if your code is written according to current standards. For instance, classes defined using the upcoming class syntax will be fully interchangeable with classes defined with the current constructor.prototype syntax. So in five years a developer can rewrite classes into the ES6 format on a file-by-file basis without breaking anything -- assuming, of course, that you also have good unit tests.

Second, avoid trendy JavaScript app frameworks, especially if they change the way you code your app. Backbone was all the rage, then SproutCore and Ember were, and now Angular's the framework everyone loves to promote. They may be useful, but they also have something in common: they often break apps and require code changes when new versions come out and their longevity is questionable. I recently updated an Angular 1.1 app to 1.2, and quite a bit had to be rewritten. Likewise, going from Backbone 2 to 3 requires a lot of HTML changes. Standards are slow-moving for a reason, but these frameworks move fast and things breaking periodically are the cost.

Plus, new official standards often leave old frameworks obsolete, and when that happens those frameworks either mutate (with breaking changes) or are left behind. You know what's going to happen to all the world's competing promise libraries once ECMAScript 6 is ratified and all browsers support its standardized Promise class? They'll become obsolete and their developers will stop updating them. If you picked the right framework your code might adapt well enough, and if you guessed poorly you'll be looking at a major refactoring.

So if you're thinking of adopting a third-party library or framework, ask yourself how hard it'll be to remove in the future. If it's a framework like Angular that can't ever be removed without rebuilding your app from scratch, that's a good sign it can't be used in a 40-year architecture. If it's a third-party calendar widget that you abstracted with some custom middleware, replacing it would take a few hours.

Third, give it a good, clean app structure. Even if you aren't using an app framework you can still take advantage of developer tools, build scripts, and a good clean design. I'm personally a fan of Closure Toolkit's dependency management because it's lightweight and its overhead is completely removed upon building your app. LessCSS and SCSS are also great tools for organizing your stylesheets and building out standards-based CSS stylesheets for release.

You can also organize your own code into single-use classes with an MVC structure. That will make it much easier to come back several years into the future and know what you were thinking when you wrote something, and to replace only those parts that need it.

You should also follow the W3C's advice and keep presentational information completely out of your HTML. (That includes cheats like giving elements presentational class names, like "big-green-text" and "two-columns-wide".) If your HTML is semantic and CSS is presentational, it'll be much easier to maintain and adapt it to new platforms in the future. It'll also easier to add support for specialized browsers for blind or disabled people.

Fourth, automate your tests and make sure you have near-full coverage. Write unit tests for every class, whether server-side or in JavaScript. On the front end, make sure each class performs according to its spec in every supported browser. Automate these tests from your build bot for every commit. This is important for both longevity and reliability, since you can catch bugs early even when current browsers obscure them. Both Jasmine and Google Closure's JSUnit based testing frameworks are good.

You'll also want to run full UI functional tests, which Selenium/WebDriver are good at. Basically, you write a program that steps through your UI and uses it as if a person were testing it. Wire those up to the build bot as well.

Lastly, as others have mentioned your data is king. Do think through your data storage model and make sure it's built to last. Make sure that your data schema is solid, and make sure that's tested thoroughly as well on every commit. And make sure your server architecture is scalable. This is even more important than anything you do on the front end.

Answer 5

Leaving aside the issues of the unreasonable expectations of your client and focusing on the design issues, I wouldn't go as far as 40 years, but the problem you seem to have, of long-term evolvability, is precisely what REST was created for. By that I really mean REST as an architecture style, not the buzzword driven development that's so commonly associated with the term these days.

To some extent, people get REST wrong because I failed to include enough detail on media type design within my dissertation. That’s because I ran out of time, not because I thought it was any less important than the other aspects of REST. Likewise, I suspect a lot of people get it wrong because they read only the Wikipedia entry on the subject, which is not based on authoritative sources.

However, I think most people just make the mistake that it should be simple to design simple things. In reality, the effort required to design something is inversely proportional to the simplicity of the result. As architectural styles go, REST is very simple.

REST is software design on the scale of decades: every detail is intended to promote software longevity and independent evolution.

http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven#comment-724

You mentioned you intend to use a RESTful interface. That comment by itself suggests you should do some serious research into that and try to understand what REST really is about. You're probably associating it merely with the mapping of HTTP methods to CRUD operations that most people think REST is, but it has nothing to do with that.

Think of REST as a formalization of the architecture of the web itself. In one way or the other, there are many parts of the web written a decade ago or more that are still available and can be used with a client made today, so we got something right in that department. It will be a lot of work, I guarantee you, because doing REST right is hard, but the long-term benefits are worth it.

Answer 6

After I've read the question and the other (very well thought out) answers, I've thought that I'll leave my two cents as well. Note: I don't have to maintain any really old application/software at all. What I use as reference is my own work-in-progress hobby web app that grabs some open government data, parses and displays it in different formats. The app started as a project where I wasn't alone and where the gov just announced that it will offer this data somewhen, somehow. So it was clear that lots of things will change over time. And they did. What I learned from it:

• Split things up in mini-applications. Each fully able to fulfill its task on its own. This makes switching out a single piece very fast, very secure and very easy. And when you have to go back in, it's not really hard to get around why things happen and how you they happen. If you or someone else will have to change something later on, it's easier to replace a single part than a whole set of things.
• Get a solid constant middleware/-layer that is used for communication between the different parts. In this case I used JSON, but XML, ini or similar standards would be fine as well. It's easy to repeat and can be transformed into nearly anything. All are proven standards that will survive quite some time. Even if the underlying data and/or storage model will change. Each of the apps can use their own data storage for its specific task. This makes the amount of scanned data for a task smaller, therefore easier to handle and maintain and is easier exchangeable.
• Don't worry about programming language decisions. Those will change over time. Just make sure that you use the language that you're comfortable with or that fits a task best.
• Make sure that your data storage is "horizontally scaleable" and that it's easy to plug in additional storage modules.
• Get a common point (in my case it's URIs) where the mini-apps are called and/or exchange data.

Summed up: The thing I'd care most about is the separation of concerns and exchange-ability of parts that are assigned for tasks. You simply know that in 40 years (even in 5 or 10) hardware, interfaces, storage, etc. will change a lot. And later developers will have to react on those changes and exchange parts of your application, be it the data container or parts of the User Interface.

Answer 7

To make things as "future-proof" as possible, plan for change. That is, try your hardest not to optimize for anything other than the ability to easily change. So no normalization, no strict validation, and loose coupling galore.

• Use major open-source technologies. For data, closed-source systems are a major source of risk as one can't plan on which companies will go under or change strategies, taking with it all access to the data. Also, minor open-source projects without a vibrant community are also more likely to lose support.

• Use a schema-less NoSQL database. The sorts of unstructured data being used is almost straight out of the textbook for a document store like MongoDB. Traditional relational databases and their normalization is good when you know how your data is going to be structured, but that's really a fiction, especially here. The costs of changing schema in a RDBS just get larger and larger as the system expands. Know that whatever structure is chosen now is going to end up changing.

• Decouple the system heavily using widely accepted standards. Breaking up all data access and mutation into web services is one step towards this. Combining that with message queues for submitting changes and such will help individual parts of the system change languages or technologies over time.

Answer 8

OK, so I'm going to say some things here which are probably going to be pretty unpopular, but stick with me here.

As this is your first project where the data and/or the application is supposed to last for 20+ years, and you are the one leading the project, you need to take a step back and think about what the odds of this project succeeding are. Because they are basically next to zero.

You need to spend a huge amount of time focusing on the database design and getting that right. For this project to succeed you need to bring a data architect into the project, and sooner rather that later. Without someone who is experienced in database design, and who is well practiced in looking forward to how the data can be used in the future the odds of the data still being useful after 5 years much less 40 years is slim to none.

Expecting two people (one of whom has the title of jr. dev) to build something from scratch which is expected to last 40 years, is probably not going to succeed. There should be a team of people many of who are experienced in working with large projects like this who are working on the data design, the API design and the application design. Something like this isn't a 2 person project.

Wanting to tie the project to something like Drupal shows why the project needs people who are used to working on these sorts of projects. You don't want to tie the application to anything that may go out of style within just a few years. If you did, finding someone to work on the system in 5-10 years could get very difficult very quickly.

I would take this advice to management and explain to them that you need to get more senior people on the project. Otherwise the project is doomed to fail, and you'll probably end up being the one that takes the blame for it.

Answer 9

The application need not survive 40 years without any evolution. But, because it would or should be built from scratch, it could still be 'functioning'.

What is the most key thing is the 'data architecture' that allows for some stability and governance as well as extensible.

We've designed data architecture and taxonomy that could almost survive the end of humanity but still be extensible. You've got find a true DATA TAXONOMY / DATA ARCHITECTURE person to do this for you.

Answer 10

The key here is to focus on the database (as several above have said). This needs to be coherent and completely describe the operation. It needs to grow with the operation as it changes. If it is not easy to change then it will get out of date and that is the kiss of death. The rest is relatively less important.

I disagree with those above who suggest normalisation is not important, though there are always cases where the current systems are not up to the job. In these cases denormalise but ensure the database handles the extra writes/ changes as part of an atomic transaction. That way you can effectively ignore the problem till you can fix it.

The company I worked for before retirement is running a system that was written from scratch and has grown almost continuously for 25 years, and covers virtually all aspects of a medium retailer. Aspects of this system that I feel were important are:

• Integration is vital.
• The database needs to be as correct and understandable to both IT and other staff, so an almost paranoid emphasis on naming is required. We have tables of defined mnemonics which are then combined to make up table and field names and all “codes” were likewise named as constants and stored in a EAV table structure.
• We encapsulated business logic into database triggers. This is painful at first and requires additional work to transmit error messages to clients and to allow triggers to be changed flexibly without locking the whole table on some systems, but this quickly becomes a massive time saver, and forces the database to be much more correct than otherwise.
• Assume you will keep at least reference tables (ideally all but the fastest and least important transactions) for the life of the system, even when “deleted” so your references are correct.
• Because of the above ensure any unique identifiers and transaction numbers are sized for the long term. (I originally jokingly suggested that they needed to last until I retired).

Answer 11

I would suggest using event sourcing and command and query responsibility segregation. This is mainly because the only thing that you can be sure about is the events that already appeared. New types of events might come. So even if you put heavy thoughts on a model, it is sure that this will get outdated. Migrating all data with every release might not be feasible. So the best is to have a model that fits your current needs and which is calculated from already recorded events every time you need it and then to pass out events that are computed from the current state of that model.

Also have testing in mind. If the application is in use in ten years from now, testers have to make sure that it is still doing what its supposed to do. So make integration testing your application as easy as possible.