Timeline for How to tackle synchronizing large amounts of data across services in an efficient way?
Current License: CC BY-SA 4.0
18 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Aug 12 at 8:48 | review | Close votes | |||
| Aug 17 at 3:08 | |||||
| Aug 12 at 8:25 | history | protected | gnat | ||
| Aug 11 at 10:02 | history | bumped | CommunityBot | This question has answers that may be good or bad; the system has marked it active so that they can be reviewed. | |
| Jul 12 at 9:32 | answer | added | Flater | timeline score: -2 | |
| Jul 11 at 22:02 | answer | added | J_H | timeline score: 0 | |
| Jul 11 at 22:02 | answer | added | Steve | timeline score: 1 | |
| Jul 11 at 20:25 | comment | added | SeeIfIDont | (For clarification: The differences per store brand is basically: some fields are fed from different places in the source system; some products only show up in some stores. The model is not really different at all between store brands and the services are all built multi-tenant style to handle everything across all brands. The number of products changes, but at around a 5-10% difference across the entire repertoire. So it's not a good leverage point.) | |
| Jul 11 at 20:22 | comment | added | SeeIfIDont | @user1937198 I certainly could, but that doesn't help us in the current architecture. All stores consume the same shape of data through the same services, the same frontends and so on. Outsourcing the differing data would unfortunately not save enough space to make a difference - I think even if it lopped off 70% it wouldn't make me less nervous about the things that I'm nervous about, being how data travels between steps. | |
| Jul 11 at 20:19 | comment | added | user1937198 | For the per store brand metadata, could you for example have a dedicated service for just that metadata which only services interested in that store would subscribe to? | |
| Jul 11 at 20:11 | comment | added | SeeIfIDont | @user1937198 The product catalog itself is 99% shared information, including some metadata that is different per store brand (descriptions, date launched), stored in the source system. And the product repertoire is heavily overlapping, with maybe 5-10% different between store brands. | |
| Jul 11 at 20:10 | comment | added | SeeIfIDont | @user1937198 Several of the downstream systems don't care about any product information at all as such. Several downstream systems would produce feeds for limited subsets. But the base unit of information about each product is reasonably heavy, and for it to not be explicitly coupled to their needs, I image you have to store as much information as possible in the event. (Or you have to stuff all information in a separate place and then tell the other systems "go fish" for each individual product - but doing that for 200K products sounds like a very bad N+1 style inefficiency.) | |
| Jul 11 at 20:07 | comment | added | SeeIfIDont | @freakish It's not 200k messages on the event bus as such - it's handling 200k things, and the various knock-on effects that can have for the various services as they process the message. What that is varies by service, but I imagine a scenario where it involves going to that service's data store to figure out if it's worth handling. If it takes more than half a second per message, you're spending a day, and so have to be clever about handling the messages lest you take your own arm off. | |
| Jul 11 at 19:20 | comment | added | user1937198 | One question that may or may not be useful depending on your system: do you need a single product catalog? Can you slice off parts of the definition of a product entity or subsets of your catalog that are only relevant to some of the downstream systems? | |
| Jul 11 at 19:17 | review | Close votes | |||
| Jul 16 at 3:03 | |||||
| Jul 11 at 19:15 | comment | added | freakish | 200k messages per day is not heavy at all. That's like 3 messages per second on average, it's nothing. Some people report Kafka capable of handling 1mln messages per second. Handling 10k per second is common, with pretty much any message broker. | |
| Jul 11 at 19:12 | comment | added | user1937198 | For problem c, what are the sorts of downstream systems that need this information? | |
| S Jul 11 at 18:56 | review | First questions | |||
| Jul 11 at 22:04 | |||||
| S Jul 11 at 18:56 | history | asked | SeeIfIDont | CC BY-SA 4.0 |