Hopefully the title is not painfully obscure, but as it indicates, I'm really looking for advice on my architecture proposal, as the current architecture is subject to concurrency issues, performance issues and dictated by third party software.
The platform is essentially a naïve Micro-Service pattern followed by being accessed in a Multi-Tenant fashion.
We have a core API that inserts core data, but the core data needs to be complimented by other data, in this example:
Now the idea is to insert core data, then publish to a queue, each module needs to subscribe, compliment the data and then publish back into the queue, which then is subscribed via the other modules sequentially. By that point all data is complimented giving a full set of data, which then is finally subscribed via the last module, which then does some super-uber magic and returned.
We expect volume of thousands of requests and possibly millions in due course. Performance/speed is absolutely vital but more performance. Limiting timeouts and hang-time, is also vital and can be achieved via parallel/background tasks. It's the throughput volume that is the highest priority.
- Is such architecture feasible?
- Can queuing based technology have/contain the ability to publish, subscribe the same data sequentially until that data is deemed complete? each time we receive the data compliment the data and push back i.e. publish