3

I am working at a retailer with a deep catalog of products and we are taking stock of our architecture.

Currently, we use microservices, but all product information goes to a service that is getting bigger and bigger over time. It continuously ingests delta update feeds from the source system where all product information is maintained, and also receives a full feed every night. This is ingested and updates a database, and this database is then used directly (other microservices reading from the database) or indirectly (other microservices calling a product API which reads from the database) from a large and growing number of services handling other concerns - product feeds in various formats, integrations with other systems.

We are looking to change this since microservices shouldn't read from each other's databases, but the trouble I'm running into is the practical implications of the other solutions. There are above 200K products with many more coming, and the changed information needs to make its way to the other services quickly.

(I will use the term "downstream service" here; that's just to say that it will feed off of the events produced by the catalog service. The idea is for them to be as decoupled as they can, but for them to need to consume and maintain their own database of basic information about a product (a different set for each service) at some point as part of doing what they do; at the very least they'd need to know that "part number 123" now exists or has gone up for sale, for instance.)

Problem A, data flow: The microservices approach to solving this in a decoupled way is to post the new information as an event on an event bus, which would then need to be a series of events. To keep the catalog ingesting service decoupled, it should do what it does being ignorant of what downstream services already know. Which means that every night it would post 200K messages about the products being updated, and everything downstream would have to process this. I buy into the idea of separation and every service owning the part that matters to it; this just sounds a bit heavy.

Problem B, bootstrapping: When a new downstream service comes online, how does it get access to the full catalog? Let's say that in addition to posting messages, at the time of the full update, it also serializes a full dump of all products as known to it on a blob storage somewhere, and the downstream service can fetch it as a shortcut for bootstrapping.

If a new service comes online at 11 AM and does everything right: recognizes that it is starting from scratch, immediately starts listening to updates on the event bus, puts them on an internal queue, grabs the full dump first, ingests them and then ingests everything that came in from the event bus, it would still be ignorant of the updates that happened between the time the full dump was produced (let's say at 6 AM) and the events that came in after it started listening at 11 AM.


The best workaround I am currently considering is allowing the downstream service to start listening, and also calling the catalog service and requesting a special bootstrapping dump snapshot to be written with the current data, which is then delivered via the claim check pattern through the event bus. But this does have them tightly coupled to each other, and also ties the catalog service up with doing a lot of work on behalf of the downstream service.

Problem C, architectural purity: If everything depends on downstream services ingesting a huge file produced by the catalog service, is that really less coupled? It's asynchronous and won't block the source system, sure. And we have separated the domain model used from the one used in the database, sure. But is this really the best we can do?

So the question is: What are appropriate solutions to these problem that minimize the issues and do not require us to "cut against the grain" of microservices?

I don't think I'm the first to have these thoughts and issues, but I'm having a lot of trouble finding the material that no doubt exists about these kinds of integration issues. This is the first project I'm working on that is trying to fully embrace microservices, and I'm still lacking for experience in all areas, and know which "old" solutions to reach for but not the "new". If there are any books or materials that are entirely about going into problems like this, I would be happy to hear about them.

Improve this question
9
  • For problem c, what are the sorts of downstream systems that need this information?
    – user1937198
    Commented Jul 11 at 19:12
  • 1
    200k messages per day is not heavy at all. That's like 3 messages per second on average, it's nothing. Some people report Kafka capable of handling 1mln messages per second. Handling 10k per second is common, with pretty much any message broker.
    – freakish
    Commented Jul 11 at 19:15
  • One question that may or may not be useful depending on your system: do you need a single product catalog? Can you slice off parts of the definition of a product entity or subsets of your catalog that are only relevant to some of the downstream systems?
    – user1937198
    Commented Jul 11 at 19:20
  • 1
    @freakish It's not 200k messages on the event bus as such - it's handling 200k things, and the various knock-on effects that can have for the various services as they process the message. What that is varies by service, but I imagine a scenario where it involves going to that service's data store to figure out if it's worth handling. If it takes more than half a second per message, you're spending a day, and so have to be clever about handling the messages lest you take your own arm off.
    – SeeIfIDont
    Commented Jul 11 at 20:07
  • @user1937198 Several of the downstream systems don't care about any product information at all as such. Several downstream systems would produce feeds for limited subsets. But the base unit of information about each product is reasonably heavy, and for it to not be explicitly coupled to their needs, I image you have to store as much information as possible in the event. (Or you have to stuff all information in a separate place and then tell the other systems "go fish" for each individual product - but doing that for 200K products sounds like a very bad N+1 style inefficiency.)
    – SeeIfIDont
    Commented Jul 11 at 20:10

3 Answers 3

Reset to default
1

do not require us to "cut against the grain" of microservices?

The grain of microservices is one microservice per independent development team in your organisation.

An independent development team being responsible for a basically free-standing application which is independently deployable, that may have some periodic data flows to or from other applications, but doesn't require constant availability of all the others to do anything useful for itself.

I have the strong suspicion that you're actually overseeing all these microservices yourself (or as part of a single team), and that they are not working independently at all, and therefore a pretentious division is occuring in the code and the architecture which doesn't reflect the reality of mutual dependence.

Well-designed applications of non-trivial size, have their layers and modules, but it isn't normal to try to completely separate things which any one person can comfortably handle in full, because all that results is inefficiency and unnecessary complexity.

We are looking to change this since microservices shouldn't read from each other's databases

No. But the lesson here is not to unnecessarily divide or duplicate a shared database in order to indulge a charade of decoupling (and then create a massive bulk flow between them), but that a mutual and real-time sharing of data strongly implies that multiple microservices have been created inappropriately where there should be just a single one.

Improve this answer
14
  • You are correct that all of these microservices are maintained and written by a single development team. Which is not to say that the business aspects these services take care of are from the same place in the organization. The reason we're thinking that the service needs to be divided or reworked is because it acts as the source of truth for multiple other services. Those services do represent their own "bounded contexts", within which the domain model is different, which is often claimed as a strong reason to have separate services in the first place.
    – SeeIfIDont
    Commented Jul 11 at 22:16
  • @SeeIfIDont, obviously we don't have a full view of the situation, but it strongly sounds to me like you're not creating separate services, but ultimately just reconfiguring a direct connection into a shared store (which if it's a database engine, is designed to stay up and available indefinitely, even through many kinds of maintenance activity) into a much more complicated and bespoke arrangement for real-time distribution of duplicated data from that shared store - a duplication whose scale and style poses the engineering challenges you're now concerned about.
    – Steve
    Commented Jul 11 at 22:58
  • To take concrete examples, right now there are three kinds of consumers. #1 is a separate project, co-deployed with the service, which exports various feeds. Right now, they go down into the same database and query up what each of them need all the time. #2 is completely separate services that for various reasons need to act on product updates, usually to update external systems with various details. #3 is an also co-located product API, that vends product information and which some other services call for various reasons, like historical product slug information.
    – SeeIfIDont
    Commented Jul 11 at 23:22
  • 1
    @SeeIfIDont, because what they're advising against implicitly, is reaching directly into data stores which belong to other development teams. But everything here is all within your own menagerie. Moreover, the database engine itself could be pressed into functioning as a kind of Janus-faced storage service - storing data in one schema for the main product service, and maintaining (through triggers or whatever) duplicate storage schemas for the services which consume the product data. (1/2)
    – Steve
    Commented Jul 12 at 0:13
  • 1
    @Flater, "team-centric monoliths" is probably a good term for it. Also, I'm not arguing for completely disjoint functionality to be put into bed together - so in that sense, a team could end up looking after two or more services which individually impose only a part-burden and whose designs don't relate to one another at all. But with the OP we're talking about so-called separate services that are somehow interdependent, and who is not talking about a specific problem he's solving, but simply about "architectural purity". Multiple related services per person is the deviation, not the ideal.
    – Steve
    Commented Jul 12 at 6:31
0

background

Having multiple micro-services query a central Source of Truth database is not a problem. They must not write to it, although reading is fine.

It is essential that a service should be able to come up (e.g. after power fail) when central DB is down. Services may choose to read rows from central DB and locally cache them, to support availability goals. Frontend the DB with a thin CRUD API layer if you like.

Revving a central schema is an area of concern. Fix it with a VIEW. (Or with multiple VIEWs.)

  • Service A reads the product table through a VIEW named product_a, and so on. Rev the underlying tables without altering the shape of the VIEW. Rev the VIEW when service A is ready for that.
  • All services read the product_v2 VIEW, which is supported by an underlying product table. CREATE VIEW product_v3 when ready, and one by one services can choose to switch over to that.

efficient deltas

Your initial Problem A setup sounds sensible.

Which means that every night it would post 200K messages about the products being updated, and everything downstream would have to process this.

I disagree. We could break the 200K items into one or more good sized chunks, each with a unique ID, and publish those, along with metadata. The IDs could be RDBMS column values, filenames, or amazon S3 file object paths. The metadata could include {begin, end} timestamps, hashes of a bunch of attributes, and product categories. All of this can be used to let downstream consumers quickly decide that a given SKU or a whole product category is of no interest today and should be quickly ignored.

Or follow the related approach of publishing just a single message each day, to announce "fresh data is ready!". And then each downstream service can send some efficient tuned SQL query to the database, retrieving exactly the SKUs of interest to that service.

use a database

Problem B, bootstrapping: ... new ... service ..., how does it get access to the full catalog?

With a SQL query. I don't understand why this is even an issue. You're describing the core use case that relational databases were designed for. Query for that service's product category, or for "recent" updates, or something else related to the business problem you're trying to solve.

Snapshots will always be out-of-date by some amount. Prefer live SELECT statements against a frequently updated DB. Use continuous DB replication technology if you're concerned about "too many services" hitting a single database.

Improve this answer
7
  • 1
    This is very interesting. My natural inclination is to say that as long as the ownership is clear, letting other parties have limited/readonly access to the database is not an issue. But most thinking I can find on microservices lists this as a thing to not do, since that couples your service to the other service. (At the very least to the database schema.) Then again, again, I have also heard people say that pure events should only ever have the domain key in it, and then let you fetch them elsewhere, which goes against the "services should not call other services". I am confused.
    – SeeIfIDont
    Commented Jul 11 at 22:20
  • If you're writing "a service", it is OK for that service to depend on DB uptime when it responds to clients. If you're writing "a micro-service", it is essential that "dead DB" shall not impact your availability to clients. You must be able to ride out service outages, whether from peer micro-services or from some central DB. This tends to motivate replication of selected rows into your own cache, so after reboots you can still guarantee uptime of your micro-service. And to do that efficiently, you're probably going to need timestamps on most rows. Then CREATE INDEX on each timestamp column.
    – J_H
    Commented Jul 11 at 22:32
  • @SeeIfIDont, on "coupling to the database schema", there's no style of programming in which the processing isn't coupled to the shape and structure of the data being processed. If you're not coupled to the database schema itself, then you're just coupled to the shape of whatever interface sits in front of it, and that interface is in turn coupled directly to the schema.
    – Steve
    Commented Jul 11 at 23:03
  • @Steve that's true, but if I send the information as events, I have a separate contract that I can version separately from the way it's stored in the database. I guess views could also provide that separation.
    – SeeIfIDont
    Commented Jul 11 at 23:18
  • 1
    @SeeIfIDont, as you say, there are various forms of indirection available in the database engine, and can be usually be updated safely on the fly without taking anything offline. Remembering that the purpose of microservices is to improve reliability and manageability, you'd have to ask yourself whether a solution divided into separate services and independently available but depending on complicated bespoke development for duplicating and distributing bulk data and synchronising after interruptions, would really be more reliable and manageable in your circumstances than just sharing the data.
    – Steve
    Commented Jul 11 at 23:45
-2

We are looking to change this since microservices shouldn't read from each other's databases

I feel like you've misinterpreted this advice. It is true that one service shouldn't directly access another service's data store, but that doesn't mean that one service can't fetch data via another service.

The key issue in the first statement is with directly accessing the data store. If this was instead routed to the owning service, who in turn would resolve your request by looking into its own data store, that is perfectly fine.

What you're doing is copying your product database all over the place. That's not helping anyone, and it effectively undoes the benefits of having a microservice represent a specific bounded context.

the changed information needs to make its way to the other services quickly.

Initially, I wanted to counter that claim, but in reality you may have designed your system in a way that it expects these massive datasets to be pushed around your ecosystem. If so, you need to take a step back and revisit that design decision, instead of trying to dig deeper into it.

You have a product catalog whose data content gets frequently updated. You need your downstream services to be aware that an update happened. That's a valid use case for some kind of event hub to create that kind of cross-service awareness.

However, that event doesn't need to carry the entire dataset. It just needs to carry a description of the event and the identifiers with which you can fetch the resource from the service. Your event wouldn't be:

Okay so product 123 was updated at [timestamp]. It is now a red fleem with 13 different trim levels, first of which is trim level yupyup, which entails ...

Your event would be:

Hey product 123 was updated at [timestamp]

If the consumer cares, then they can go to the originating microservice e.g. /api/products/123 to fetch all the product's data.

Your data isn't stale (the request was just fired), you wouldn't have issues with new services nor service outages (you can just get the current state), you wouldn't be duplicating your data all over the place, ...

You don't need to engage with all the problems you're pointing out in your question. It all relates back to having decided to create some kind of event-drivrn distributed database, which I think is borne of a misinterpretation of the microservice advice to not directly access another service's datastore.

Improve this answer
15
  • So your suggestion is rather than have 200k events to process, he needs a service that can handle 200k * N services requests for updates?
    – user1937198
    Commented Jul 12 at 9:37
  • If the majority of clients end up fetching the majority of the data in order to avoid outages, how is that not just duplicating the data all over the place?
    – user1937198
    Commented Jul 12 at 9:38
  • @user1937198: I'm not sure which of the myriad contemporary approaches to designing these kinds of systems you want me to respond with. Your question vastly oversimplifies the scenario
    – Flater
    Commented Jul 12 at 9:39
  • @user1937198: Other services don't need to store the data. They access it as they need it. Duplicating data creates multiple sources of truth, which opens the door to sources contradicting one another and having to work around countless data synchronization issues. All that effort and constant source of bugs and support issues can be avoided, and the load of requests can be addressed vua the appropriate tech stack. What I'm proposing is not some arcane design philosophy, it is the main point of a microservice ecosystem.
    – Flater
    Commented Jul 12 at 9:40
  • 1
    @RikD: There's definitely some compromise here, I'm not a hardliner. OP mentioned price updates, yeah it's reasonable for the event to include a product ID and its new price. But sending the entire product update is several bridges too far. If all that information is needed by everyone, then there is no functional "products microservice".
    – Flater
    Commented Jul 12 at 23:32

Not the answer you're looking for? Browse other questions tagged or ask your own question.