I am trying to improve the architecture of a system where data flows between 3 systems. Every minute, there are thousands of items generated on App 1, which are sent to Midpoint in the following way:
- App 1 sends items (through REST API) to Midpoint
- This happens on an "as-and-when" basis: App 1 will keep sending items when needed
- Midpoint receives items and saves it into its database
- Every minute, a cron job runs that takes items from this database, performs business logic and sends them (through REST API) to App 2
- App 2 performs business logic and sends back a response to Midpoint, which uses ActiveMQ to send the response back to App 1
There is usually 1000+ items being sent from App 1 to Midpoint every minute. As a result, Midpoint is under too much load and very slow. Once an item gets sent from App 1 to Midpoint, its response can take up to 30 minutes to travel through the flow and come back to App 1 (Midpoint -> App2 -> Midpoint -> App1).
Are there better solutions out there or design philosophies for this kind of task? There must be a way to make this task more efficient and fast. How are all the big companies handling it with billions of transactions?