Microservices have technical and social aspects:
- each microservice can be developed and deployed independently
- each microservice can scale independently
For each of these aspects – development, deployment, scaling – it took some time until microservices became a sensible option.
Most problems then and now are simply not so big that scaling would be a concern. And long ago, the cheapest way to scale was not to build distributed systems but to build bigger machines. What has changed since then: now we are solving many more problems with computers, in particular internet-related stuff, and quite a few problems have performance requirements that cannot be satisfied by a single blade server. Bigger machines are not an option beyond some point due to physical limits. So distributed systems it is.
As a sub-point of this, distributed systems can be attractive because they are potentially more highly-available. This doesn't imply microservices as a monolithic architecture can be made HA as well, but microservices are a popular way to build highly available systems.
Microservices aren't necessary for a system to scale, but it allows different parts of a system to scale independently. For example, some web app might need a different number of frontend servers, database replicas, and worker nodes for optimally cost-effective performance.
Software development used to be release-oriented: requirements go in on one side of the process, developers do their thing, and some time later a release emerges on the other side. Such a view causes difficulties if requirements are uncertain and need to be tested by putting them into practice, if there is ongoing maintenance without clear releases, or if the project is simply very big so that it's ineffective and too risky to release all components together.
The dot-com bubble shifted priorities towards minimizing time to market. “Rapid Application Development” was a big thing. Agile techniques emphasized iteration and early deployment. One of the most influential agile techniques were things like automated unit testing, continuous integration, and (much later) continuous deployment. DevOps builds upon these agile ideas and suggests a strong focus on making deployments automated.
All of this is relevant for a service architecture because services can be modified and deployed independently so that changes reach production faster. That involves a lot more total deployments than for a monolithic architecture, so that automated build and deployment systems are a prerequisite. Independent development also means that different services can be built with different technology stacks, as long as the services can communicate via some common interface.
Containers are not necessary for microservices, but containers simplify system administration and container images make automated deployment a lot easier. Similarly, “the cloud” is not necessary for microservices. But a microservice system does need a cluster of machines to run on, and cloud platforms are an easy low-capex way to get that cluster.
Microservices represent the current best practice for developing large systems that need independent development, deployment, and scaling. This is based on the currently known technologies and development techniques.
However, microservices are not the first occurrence of service-based architectures. Before that, service-oriented architectures (SOA) were already popular, especially in an enterprise context. SOA and microservices have overlapping goals and techniques, but SOA tends to be about integration of different services over a common message bus/queue. Microservices are an evolution of SOA, but may favour more fine-grained services and don't necessarily have a centralized message queue.
Microservices will also not be the last occurrence of these ideas. Now that the microservice hype is dying down they are becoming an ordinary tool in the system architect's toolbox, but our technologies and techniques continue to evolve. Interesting directions include scaling down with FaaS, and moving computations to the data with edge computing.