1

I've been tasked with re-designing a system that is, for lack of a better term, complete spaghetti. For various reasons I feel that the system is well-suited for microservices. At a high level I feel that every service should...

  • Function standalone
  • Not have direct dependencies on other services
  • Have a defined API to interface with the service
    • The API may be strongly typed but it would essentially boil down to asynchronously sending a command to the endpoint associated with that service and either RESTful HTTP calls or gRPC for the more request/response type functions.
  • Be focused on a single problem domain
  • Own its data
  • Pass the chaos monkey test (everything still working if a service goes down randomly)

I also feel that the system should be able to easily grow/shrink over time. Using events, eventual consistency, and allowing for a sense of data duplication between different services makes sense to me. Where I'm struggling deals with adding a new service to the system. In order to do that how does the system handle

  1. The new service catching up to what has happened in the system while it didn't exist
  2. Existing services knowing about what events/commands that new service supports

Existing thought is if you already have two services (A and B) before introducing C, and A needs to do things when events from C happen, you end up creating a new version of A along with adding C to the mix. Then a mechanism is needed for somebody modifying the system to go "service C is now something that can be installed and, by the way, a new version of A exists.

I have no clue how to get service C to catch up to what has happened while it didn't exist.

Example

There is an inventory service that manages what is in inventory. Also, a users service exists so we know who took things out of inventory and put things into inventory. As users are added/removed to the system the inventory service consumes those events to keep its storage mechanism up to date. That's so when the system queries the inventory service it doesn't have to do direct service communication to get first name/last name information for the view.

Later we add an automated way to pick things out of inventory. This machine will publish an Inventory Picked event that the existing inventory service needs to know about so it can remove that item picked from inventory. On top of that, this new service needs to catch up to where the inventory service is since things have been added and removed from inventory while this new service didn't exist.

Does the inventory service need a new version at the time this picker service is introduced (because there is a new event it cares about that didn't exist before)? How does the inventory service get caught up without having a direct dependency on the inventory service?

5
  • I think this question could be improved by a concrete example.
    – JonasH
    Aug 16, 2021 at 15:05
  • @JonasH I'll find some time either today or tomorrow to provide a concrete example in the question. Thank you!
    – Logan K
    Aug 16, 2021 at 19:37
  • Are you planning to use kafka?
    – Darem
    Aug 17, 2021 at 5:21
  • @Darem I'm looking into Kafka but my hands might be tied and will end up using RabbitMQ
    – Logan K
    Aug 17, 2021 at 11:31
  • You can have a look at the concept of "schema registries" that could also work with RabbitMQ. Especially RabbitMQ got "streams" in the latest version. Therefore both of your problems would be solved.
    – Darem
    Aug 18, 2021 at 6:08

2 Answers 2

1

At a high level I feel that every service should [..] Not have direct dependencies on other services

That is not necessarily a realistic expectation.

If we're talking functional dependency, microservices still depend on each other in the sense that service A may call service B, and B's absence may render A unable to perform its own job. For example, you can't place an order (A) when the warehouse service can't confirm that the item is in stock (B). That's not a technical constraint, but rather a business decision for A to depend on B's information.

If we're talking runtime dependency, i.e. the order and warehouse service are a single app and live/die together, this is the sort of dependency that microservices avoid.

At a high level I feel that every service should [..] Pass the chaos monkey test (everything still working if a service goes down randomly)

See the above point. Just because other services remain online does not necessarily mean that they will work, when the downed service has been decided to be essential to the other services' work.

As a very clear example, if the identity provider goes down, you can expect many apps to stop functioning simply because there is no way to authenticate your users anymore.
Maybe an identity provider is a fringe example as a microservice, but similar examlpes can be made for any service in the right context. E.g. you cannot schedule a meeting (appointment service) if the room's availability (room service) or invitees' identity (people service) cannot be confirmed.

Where I'm struggling deals with adding a new service to the system. In order to do that how does the system handle [..] The new service catching up to what has happened in the system while it didn't exist

This conflicts with your earlier "owns its own data" constraint.

For the service to be able to catch up what happened in the past, this implies that there is some sort of central historical data storage that confirms what the system has done/recorded.

This directly contradicts the notion that a microservice architecture decentralizes your data storage, since every service stores its own data and is the sole owner/accessor of their own data storage.

That is to say, it should not be centralized information that just anyone accesses. However, you can shift this into something more microservicey: a historian service. This services is a microservice in and of itself, which is tasked with storing all necessary history, and serving that information to requesters.
Your new service then simply connects to the historian service, asking it to list events that happened in the past. On a technical level, this is no different from how any service fetches data from another service.

To be quite honest, I'm weary whether such a dependency on historical data is a good thing, or whether it's indicative of a bad design. I can't think of a valid use case where a new app somehow needs to replay the past when it didn't exist yet.
However, since I don't know the context of your applications, I'm unable to judge this as a definitively good or bad approach.

Where I'm struggling deals with adding a new service to the system. In order to do that how does the system handle [..] Existing services knowing about what events/commands that new service supports

I am unclear how this forms an obstacle.

You handle it just like how any codebase is made to handle any change to its environment: you update the code and redeploy the service. This is no different from any other change in what your application needs to be able to do.

Microservices don't avoid needing to update your code or redeploy your services. Microservices merely avoid needing to update services B, C or D when all you need to redeploy is service A.

The only other possible option, if not updating the code and redeploying, would be for a deployed service to somehow sense what's in the network and be able to figure out for itself how/if/when to use it and interact with it, without any developer implementing it manually. That's effectively an AI, which is well beyond the scope of what we're talking about here.

4
  • I just wasn't sure if having those dependencies like I would normally write them were still okay in a microservice system. Where your thing has dependencies on interfaces to other things. Is it safe to say you just need to make if your interface to some other service fails to do "the thing" that you handle that scenario accordingly (since the implementation would either be gRPC for data access and sending a command to tell that other service to do something).
    – Logan K
    Aug 17, 2021 at 22:10
  • @LoganK: "Dependency" can mean many things. Are you talking about project dependency, i.e. A references B's assembly? That is usually a no-no in a microservice architecture. You tend to find that A will write some B-inspired DTO classes to handle the data returned from requests fired at B. However, it's possible that B has exposed a small assembly (e.g. Nuget package) containing those DTOs, which is intended to be used by its consumers, and that kind of dependency is perfectly acceptable.
    – Flater
    Aug 18, 2021 at 7:36
  • I don't mean to drag this on (and this all has been VERY useful information). Your idea of an identity service going down and most other things going down. What if a scheduling service consumes UserAdded/UserDeleted type events and stores only what it needs about users so it can perform its own authentication without needing the RPC to the identity service? Might be a dumb question or going down a rabbit hole. If so just wave me off. Thanks again!
    – Logan K
    Aug 18, 2021 at 12:48
  • @LoganK: If another service has its own validation, then the identity provider is no longer your sole authority. Which means you've built the same thing twice, which is what you tend to want to avoid in good practice development. The point of microservices is that each service has its own responsibility and it should be the sole service acting on that responsibility. Otherwise, your distinct services lose their purpose.
    – Flater
    Aug 18, 2021 at 12:51
-2

Processing older data

Your question title is currently, "adding new service into existing microservice system". Looking at the body of your question, and particularly your line "The new service catching up to what has happened in the system while it didn't exist", the title of your question would be better phrased "How can I enable the processing of older data when adding a new process?"

For this my answer is:

  • Don't use Microservices - generally avoid it in life if you can; but in this situation, don't let microservice dogma drive your problem-solving process.
  • Use Tables in a Database
  • Use Views to orchestrate data going into "processes"

When a new process is added, it has access to any new View of data it needs as input from a full potential set of persisted data.

There are many details to make this work, but that's the fundamental answer to your question.


Service Interface coordination

You also ask another discrete question: "Existing services knowing about what events/commands that new service supports".

That's either:

  • Design-time coordinated (with documentation, planning)
  • Code-time coordinated (interface contracts in code - gRPC, code-generation can work in this situation)
  • Run-time coordinated - this is generally overemphasized. You might work in teams, but you certainly communicate. Only late-bind third-party (not yet made) modules should be run-time coordinated. This doesn't seem to apply to your case.
10
  • 1
    It's not the answer.
    – user253751
    Aug 16, 2021 at 15:00
  • This answer is rooted in an absence of knowledge on how to approach this problem with a microservice architecture, as opposed to being an authoritative answer on why a microservice architecture would allegedly not work for the problem at hand.
    – Flater
    Aug 18, 2021 at 12:58
  • @Flater Microservices don't have all the answers. Service Oriented Architecture was around before, and covers way more patterns. Microservices is certainly a popular sub-set of Service oriented architecture, that needed to be able to shine on its own with a "micro distribution" emphasis. Worse still, Microservices isn't a stable definition. So I take every chance I get to malign it, so people don't try to use it as a design tool - it isn't. People should be encouraged to think from scratch about the conceptual domain first - problem-solve. Sep 21, 2021 at 13:37
  • @user253751 fixed Sep 21, 2021 at 13:38
  • @KindContributor: I did not claim microservices have all the answers for everything. But your advice, i.e. "don't use microservices" isn't particularly nuanced. The question establishes an existing microservice system. Suggesting that the only solution is overhauling the entire existing microservice approach, based on an unsubstantiated "don't use microservices, use tables and views" claim, is disproportionate and suggests that the advice comes from a lack of knowing how to tackle this in the existing system, rather than knowing for a fact that the existing system cannot reasonably solve this.
    – Flater
    Sep 21, 2021 at 13:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.