ASP.NET deployment/maintenance best practices

Question

I have been in the web development industry for around 5 years now, always working in an open source environment. Mostly apache, mysql, and php with a little bit of ruby, using git for version control. But recently took up a job where development is entirely C# ASP.NET MVC.

While I was able to pick up the language etc fairly easily, the other members of my team (with a lot more MS Development experience than me) all have a different way of thinking when it comes to publishing and deployment of the final site, and particularly future changes.

The mentality with the other developers is that once a site has been published it is final. No more changes can be made to the site, when I have asked the reasons behind this, the answers have been it is too dangerous, time consuming or difficult.

From my past experience, updating a site is simply a case of uploading the changed files, which is usually quite quick if it was a little change, or putting the site in maintenance mode while the update occurs.

We recently published an MVC Site, and the business contacted us to update some of the text and add a link to a new pdf document. The rest of my team were quick to say that this should not be done because the site is now live and should not be modified. Is there something I have missed by not being 'brought up' a Microsoft developer?

What are the argument against making modifications to a live web application in production and is this mindset unique to .NET developers?

I genuinely would like to understand this mindset and whether it is justified in a Microsoft development environment, or if this is just an older way of thinking.

NOTE: We use TFS for version control and use publish profiles to determine where the site gets deployed (UAT or Production)

Answer 1

ASP.NET MVC applications are compiled. This means that you can't just upload the changed files, like you do with a PHP website, for example. This also means that when you'll start to update the site, current users will be thrown away (lose their sessions, for example).

There is also much more to do than simply update the files: you have to handle:

• Permissions

  The new version may require a different set of permissions on the server.

• Configuration

  The new version may require the Distributed Transaction Coordinator (MSDTC) service to be installed, or may want to use a different SMTP server, or have access to Active Directory, etc. This involves changing both the configuration of the application and of the server itself.

• Dependencies

  For example, one of the issues which is often encountered by the beginners is that they keep old DLLs in /bin while adding new ones with different names: the application may still use the old ones, which creates a crazy situation where you change the code of the application, but the application behavior remains the same.

• Data

  What if database schema changed, and the application uses an ordinary SQL server instead of NoSQL? How to perform the change in the schema? How to keep the data correct during the change?

Handling the transition between an old and a new version of the application is a hard task. A few years ago, it was one of the issues, where a new version of application was ready, but took days or weeks to system administrators to deploy. DevOps addresses this issue, but requires for the developers to describe (through code or configuration) the system which will host the application.

The larger is the application, the more complicated is this task.

• For a tiny web app, copying source files to the server is enough,

• For something larger, you have to have an automated process which deals with the update process, and the rollback in case something goes wrong,

• For systems even larger, at every new version, new VMs are created and deployed, and old ones are recycled, ensuring seamless transition of the users from the old to the new version.

Compiled applications simply force/encourage to automate the process earlier.

the business contacted us to update some of the text and add a link to a new pdf document. The rest of my team were quick to say that this should not be done because the site is now live

IMO, there are no real technical reasons for this refusal; they just want to avoid doing it, because, if not automated well, the task is error prone.

What usually happens is that:

1. The application is deployed for the first time.

2. The team spends a few hours tweaking the configuration to make it work. Since the team was expected to deliver three weeks ago, everybody rushes, and nobody takes the notes of the changes.

3. The application is now up and running.

4. For a few weeks, months or years, some random people change some random stuff on the server: for example they moved a database to a different location, or the SMTP password changed, causing the changes in Web.config.

If you update the new version now, you're back to the first step, and it may take days to recover the correct configuration. Since the website is live, this should be avoided at all costs.

Answer 2

Do you have project managers, a Development Manager, or someone else other than the other developers?

It makes ZERO sense that you cannot do a deployment after go live.

Of course, these changes need to be scheduled, costed, and resourced amongst other things, but just saying "no" is nonsense.

Answer 3

The main reason for the behavior you are seeing is that these types of changes are error prone. Manually updating the application leads to things being forgotten and getting out of sync. You should still be able to do new releases (and it should be easy), just not partial patches.

Partial updates like changing some CSS or text on a live site could lead to skipping testing, test or stage environments becoming out of sync, or even, not committing code to source control. If this is a code change, you could even end up breaking the live site with no recovery plan.

In .NET because code is compiled, there can be other problems such as long load times and loss of user sessions. These issues could be mitigated my swapping to a warm environment and using out of process session state. However, unless you want to think about these type of issues and other issues specific to your application every time you want to do a deployment. Then patching live websites is a bad idea.

As your application grows in size and people come and go this can move from an inconvenience to a serious issue. So to make things safer we follow rules like: To update a live website, it must be a complete deployment, never just patched.

If you're not automating your deployments yet (which make it easy and fast to follow the rule). New deployments are usually done next to the existing version and then your website is just a pointer to the version you want (Note database schema makes this a bit more complex).

1.0.0  // Old version you can roll back to   
2.0.0 <-- IIS points here

In reality, I think that all deployments should really be automated and that includes non .NET deployments as well, for similar reasons. Once you start automating deployments you can guarantee they're consistently fast and reliable.

Answer 4

I regularly republish / redeploy quite a few ASP.NET MVC websites deployed to Amazon AWS, Windows Azure and private webservers and don't see any reasons why your team makes such a big deal of it.

However, you should design you websites in such a way so that tasks like updating texts and links should be done through the database via the website admin interface.

My point is that while changes to a live website are OK (it's called development), redeploying the application to change a text string is likely indicative of a bad design. I am not suggesting the republishing could be done without care.