I am trying to setup a multi-region (global) AWS system, where each region (which has multiple availability zones) has its own VPC. In theory these VPCs all use different portions of the private IP address space so they can communicate with each other internally. But the question is, does every VPC need to be connected to every other VPC?
More specifically, the only reason VPCs need to talk to each other in a multi-region system (in my mind) is to perform atomic database transactions (and to sync replicas). In theory, each VPC/region would have a replica of the database close at hand, to reduce latency. Then non-transaction writes would be put into a queue or something and saved eventually to some "main" or "central" database. Whereas transaction writes would necessarily (it seems) have to go to this "central" database to perform their atomic updates.
That's where the question arises. Does this sort of system need to be arranged hierarchically? Or I mean, hub and spoke? Where the hub is a central source of truth in which all transactions occur, while the spokes are simply replicas to lower latency for the non-transaction relevant data. If so, where do you put this central server, does it really matter? (i.e. can I just pick any region?). Or does the network in fact need to have connections from each region to every other region in a sort of graph? What would be the use case for that?
What is the typical setup here? Basically, it seems like the only solution to handle database transactions is to have a central source of truth, in the hub/spoke model. Is that correct? If that's the case, then I will need to pick a region and add to the security groups so that each spoke can talk to the hub, but I wouldn't need to create security groups for each thing to talk to the entire network.
The title of the question is about how you organize requests. It would seem then that you make requests to the closest region, then if there is a database transaction, make a request from that server to the central server, and do your locking, and that's it, there's no way to avoid that latency. I'm mainly asking to confirm if this is the correct architecture, or if not, what is typically done.
If an entire region goes out, then maybe it is better to not have just 1 central hub, but 2 or 3 somehow. I don't really know how that would work, but would be good to know if that's what is done.