We are contemplating breaking up our monolithic monsters into microservices based architecture progressively. We have 5 teams, each team containing 2-3 C# developers, at least 1 database developer, and 2 QA engineers. Besides the huge culture and paradigm shift going from a monolithic architecture to microservices there are also the technical challenges. I would like to ask the community for some wisdom and advice so we can avoid making the same mistakes.

Are there any good industry examples where .NET based microservices have been successfully utilized in a production system? What was the strategy for the following?

  • Org: how did you organize the .NET solutions/projects?
  • Development planning: in terms of development planning, how did you break up the work across teams? What was the overall strategy for making sure contract compliance across microservices was negotiated across teams?
  • Load balancing/routing/API gateway: what was the strategy for load balancing, redundancy, and scaling? Did you just go with a complete async architecture and used a queue for microservice communication or did you do peer-to-peer through a load balancer/API gateway? And why?
  • Test automation: how did you handle test automation. This along with continuous integration seems absolutely necessary for microservices architecture. How did you go about it?
  • Deployment: how are you deploying? One VM/microservice or one container/microservice or something entirely else? How did you handle the fact that now you have tens of database if not more considering each microservice would have its data store, what did that do to your infrastructure and how did your DBAs handled it?
  • Infrastructure: how did your infrastructure scaled with this architecture. Was it super chatty and you had to fine tune it or did the network handled it without an issues? Are self-hosted or in the cloud?
  • Monitoring: what is your monitoring strategy? How are you keeping tabs on tens if not hundreds of microservices? Is it mostly through logging and central monitoring?
  • 3
    Most of these bullet points could make a reasonable question - but as one question, this is much too broad. – Philip Kendall Jun 19 '16 at 6:08
  • 1
    With 5 teams, each team containing 2-3 C# developers, all working on the same product, you will have much bigger problems than those technical stuff. Start with one team, and one part pf the application, and try to make a microservice of it which can be used in conjunction with the remaining application. Then you can answer your questions above from that experience. – Doc Brown Jun 19 '16 at 7:12

I've worked on a number of microservice projects. Inevitably companies have taken the route because their big DB approach cant scale any further. Here are my experiences/recomenations.

Org. One solution per microservice. nuget packages for shared libs

Development. larger teams 5-6 devs one area of functionality at a time. Refactor into a interfaced service. Replace the in memory service with a client for the microservice.

Testing. integration tests using real input/output data. You want to be able to fire them against any running instance to check they are up and correct, local instances, test/uat envrioments and in memory instances for unit testing. Make sure you can test the version of the instance via a healthcheck interface or similar

Scaling. Queue based is best as is can handle multistage distributed processes. Rabbit MQ, Zero MQ, MSMQ etc. But load balanced REST services are fine for rpc style calls and can be an easy starting point.

Deployment. Octopus. db projects, self creating no-sql.dbs like Mongo. Although I think you are going the wrong route if you have multiple dbs. Instead have heavy messages which contain the data you need for the process and a few larger dbs for data storage hidden behind their own apis.

No DBAs! If you have a DBA wrting sql you are doing it wrong.

Infrastructure. No problems. Read from a queue. Do process. Write to a queue. You can get away with more than one instance per box even on cloud micro instances for small or infrequently called services

Monitoring. Health check interfaces for all services called reguarly by monitoring software and up on the big board.

Automatic failover and recovery is important, instances should spin up when needed and be stateless, so one crash doesnt keep the service off line.

The main problem isnt so much services going down as the very nature of micro services makes them robust in this respect. Its how you handle messages which cant be/havent been processed.

Use logstash or similar to keep track of message flow and work out where and what the problems are. Ensure you can re run failed messages so a fixed process can carry on where it left off.

Final note. version everything, dlls, nugets, data, interfaces. With multiple instances of multiple services and historic messages floating around it can be a major cause of problems.

  • 1
    "Is you have a DBA wrting sql you are doing it wrong.": why? I mean, aside the Agile stuff that everybody should be able to do everything, what would prevent having a dedicated DBA in a microservices environment compared to non-microservices one? – Arseni Mourzenko Jun 19 '16 at 9:08
  • Hmm difficult to explain. Usualy what I see is orgs with massive legacy sql dbs full of sprocs which they are trying.to get away from. They got in this state because of a DataBase centric approach and the DBA view that doing it on the DB is fastest. This is exactly the opposite of what microservice architecture ia trying to do. So the presence of a DBA indicates they havent escaped from.the old way of thinking – Ewan Jun 19 '16 at 9:16
  • That's not even close to true @Ewan. Our DBA sure as shit doesn't want business rules in our database. She's more concerned about being able to migrate to different servers and newer versions than writing business queries. DBA's do a lot more than write SQL if they're any good at it. This attitude of yours about DBAs shows that you haven't escaped the way of thinking. – RubberDuck Jun 19 '16 at 11:20
  • 2
    @RubberDuck: you're talking here about the system administrator side of a DBA. Ewan, on the other hand, was talking specifically about the SQL writing tasks of a DBA, while I imagined the DBAs as someone who uses their extensive knowledge of databases to help developers optimize their queries, advise on the different risks, and, in general, make their life easier when facing the complex aspects of databases. – Arseni Mourzenko Jun 19 '16 at 12:02
  • @RubberDuck yeah its "code smell" based on the limited info in the OP. you def want a DBA in the ops role doing the hard stuff. unless of course you move to a cloud db – Ewan Jun 19 '16 at 12:12

For the last 2 years we are splitting a monolith into microservices, so here are some of the things we are doing

Organisation: each service will be a solution by itself, no common projects with any other service. And we ended up the contracts being a separate solution by itself, where each version is a .nuget package.

Development: each team was working on one part of the application, for each new service we started with contract creation and then separating the service, but still keep it part of the main app/solution (so no HTTP calls yet). And in a future step we would take this service totally apart.

Routing: All our services are behind a load balancer and each service gets deployed on a few vms. We are sharing the same vm for multiple service. Going with one vm per service seemed to me as a waste of resources, because services are small while a Windows vm needs around 2G to work fine. Some services which are not related to user interaction (like sending emails/notifications) are working with queues.

Testing: Initially we thought that unit testing the service and testing the contracts compatibility between different versions of the clients and the service with Pact.Net would be enough. But we had issues when a new version of a service wasn't deployed and we worked with the previous one. All tests passed, but the whole platform wasn't working fine. So we added some high level (integration) tests on the main flows.

Deployment: All the platform is installed on a couple of vms, we are using a combination of TFS for build, AWS S3 for artifacts, Ansible for vm creation, deployment and configuration. Ansible plays a big role here, it is agentless and uses powershell remoting for connecting to windows. We stopped using xml transform of web.config and moved to Ansible templates, so we can have all configuration in Ansible files. And the good part it is free and open source, comparing to octopus. Completely new vms are used for new versions, we update services only when we have to deploy fixes.

Scaling: On such a deploy you can only scale the vms and not the service itself. So monitor your performance (CPU, RAM), number of requests you get, or even time based (like in the weekend there is less traffic) you start and stop new vms

Monitoring: We are on AWS and we have CloudWatch for time series alerts, but we are planning to go to Grafana and Prometheus (a step closer for going to docker, now with Server 2016). On the logging we are using Graylog (which is using ElastiSearch behind). It was easy to adopt it, because we are were using Log4Net with file appenders before and there is an appender especially for Graylog. You can build a lot of alerts based on it, we realized that it is actually a continuous process.

Not the answer you're looking for? Browse other questions tagged or ask your own question.