0

Yesterday, the deployment of a financial website on a client was replaced with an older version which created unexpected issues and a downtime of almost 12 hours.

We came to know that two guys from their IT are creating problems for each other and it was done by one of them, don't want to go into the details.

This could have lead to financial discrepancies and losses, but fortunately didn't.

IT guys have full rights on the server so it can not be controlled at machine/OS level.

So, is there any possibility/option available inside ASP.NET to secure or ensure integrity of the deployment?

For example what if version/hashcode of assemblies could be cross checked against database?

  • 1
    Specifically,what is it that you want? – quintumnia May 3 '17 at 11:43
  • 4
    It seems you're trying to solve a people problem (admins acting unprofessionally) with a technology solution (locking down deployed versions). That is not generally a good idea, because such locks tend to produce more problems down the road: what if the database goes down? What if you need to revert a deployment in an emergency? And who will now be responsible for marking versions as permitted in the database? That's a new people problem brewing. Instead: script the deployment as far as possible. If you can deploy (or revert) in 5 minutes at the push of a button, 12h downtime could be avoided. – amon May 3 '17 at 11:49
  • @amon 12h downtime was caused by unavailability of the core developer and source code. Support department was not able to find the problem. – bjan May 3 '17 at 12:19
1

You should add a health check page or api to your website and use green/blue deployment to prevent this type of problem.

The health check page, should return the version of the website. Ideally the versioning is done automatically on the build or deployment server and cant be changed manually.

Deploy first to the offline set of servers and run your integration/smoke tests against them, including checking the version returned by the health check webpage.

Only if these tests pass should you then switch these servers to live and take the old version off line.

Ideally your goal is to reduce the human component of deployment to clicking a single button. Automating all the tests and rollback procedures.

  • Deployment was not an issue at all, identifying the problem i.e. newer has been replaced with older, took actual time. I want to avoid such hard copy (copying older binaries/pages in the virtual path of IIS) or at least identify something is wrong – bjan May 3 '17 at 12:22
  • hence the health check and tests – Ewan May 3 '17 at 12:23
  • and what about changes in other files e.g. html pages, scripts, etc? – bjan May 3 '17 at 12:57
  • 1
    ok. you SHOULD only deploy complete websites!!! you can make the healthcheck page return the modified dates on all files. but yeah, if you are still literally copying files around by hand you need to stop and get octopus deploy or something – Ewan May 3 '17 at 13:30
0

The solution is to not allow developers to have write access to production servers on a permanent basis. If access is required because they have duties outside what is normal for a developer then give them a separate account for server administration. A fairly common auditing requirement especially for financial systems is to have a separation of duties that prevents developers who wrote the code from being the same people who can deploy the code. There are a ton of deployment automation tools available that can help you manage this, if you use TFS for source control and builds they have release management that is available or you can integrate a number of open source deployment managers into the build process.

creating a solution that limits allowed versions is a sign of massive mistrust issues with your admins that will provide minimal help in prevent bad things. A bad actor as an admin can still cause mayhem weather you so something like this or not. This would prevent accidental changes, but a better solution is to manage permission in such a way that accidentally deploying to production isn't a realistic scenario. During times when everything is working as intended and new versions are pushed according to plan such a system adds more overhead, and if updating the allowed versions gets automated to be part of the deployment process then it becomes totally useless.

You have to trust the people that are granted admin rights to not behave badly, because there isn't much you can do to prevent actual admins from doing things if they want to. It also creates a poor working environment when mistrust is the default. Additionally, you need to properly control who has admin access and ensure it is appropriate limited to those that actually need it.

  • Control Management is beyond solution/service provider's jurisdiction. I am looking for solution to protect myself, developer, solution/service provider. – bjan May 3 '17 at 13:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.