0

I am trying to improve the architecture of a system where data flows between 3 systems. Every minute, there are thousands of items generated on App 1, which are sent to Midpoint in the following way:

  1. App 1 sends items (through REST API) to Midpoint
    • This happens on an "as-and-when" basis: App 1 will keep sending items when needed
  2. Midpoint receives items and saves it into its database
    • Every minute, a cron job runs that takes items from this database, performs business logic and sends them (through REST API) to App 2
  3. App 2 performs business logic and sends back a response to Midpoint, which uses ActiveMQ to send the response back to App 1

There is usually 1000+ items being sent from App 1 to Midpoint every minute. As a result, Midpoint is under too much load and very slow. Once an item gets sent from App 1 to Midpoint, its response can take up to 30 minutes to travel through the flow and come back to App 1 (Midpoint -> App2 -> Midpoint -> App1).

Are there better solutions out there or design philosophies for this kind of task? There must be a way to make this task more efficient and fast. How are all the big companies handling it with billions of transactions?

Improve this question
5
  • What are the constraints or considerations that prevent a scale-out (or sharding) of "midpoint"? For example, are the business logic required to have access to every item within the minute? Is it required that the App2 response for each item be sent back to the same instance of Midpoint?
    – rwong
    Commented Jan 10, 2019 at 5:09
  • Is the database needed in order to make a record of what items had started and what items had finished? Is there a reliability requirement that requires the use of a database that flushes to a disk? What if an in-memory database (on a cluster of computers, possibly on different power supply / availability zone) is used?
    – rwong
    Commented Jan 10, 2019 at 5:13
  • 2
    1000 items or so per minute isn't particularly onerous. What does your profiler tell you? What about the DB profiler?
    – Telastyn
    Commented Jan 10, 2019 at 5:18
  • Midpoint is storing entries in the database to keep a log of items that are pending as well as items that have been completed. When the cron task runs in Midpoint, the app will find all item IDs in the database that have "pending" flag set to true.
    – DemCodeLines
    Commented Jan 10, 2019 at 5:19
  • @Telastyn Problem is that network requests (sending from App 1 to Midpoint and Midpoint to App 2) take time and each item is processed in the order received. This severely slows down the entire flow. As more items get added to the pending queue, the backlog just keeps increasing.
    – DemCodeLines
    Commented Jan 10, 2019 at 5:30

1 Answer 1

Reset to default
5

Since you already have a Message Queue, put the message directly on that rather than the db and horizontally scale your Midpoint and App2

App1 -REST-> Midpoint 

Midpoint (load balanced, multiple servers)
    Put message on Queue
    Write Log
    Send OK Response

App2 (multiple queue subscribers)
    Read message from Queue
    Process message (presumably slow operation)
    Write Response to Outgoing Queue

Midpoint2 (multiple queue subscribers)
    Read Outgoing queue
    Convert to App1 readable format if required
    Send to App1

Now if any layer is slow you can boot up an extra server, install a copy of the Midpoint or App2 and you have instantly increased your processing power.

The midpoint apps dont have anything that can lock or delay them, they just do their stuff and respond.

If App1 can write/read Queues you could potentially remove the midpoint completely

Improve this answer
3
  • I should have mentioned that while “App 1” and midpoint are my servers, “App 2” is an external party who I am interacting with using a REST API. The information that the API sends back is what I send back through the chain to App 1. Can we adjust the messaging queue to be able to interface with “App 2”?
    – DemCodeLines
    Commented Jan 11, 2019 at 14:35
  • Sure, App2 in this case then is just a worker process which calls the external api as part of processing the message.
    – Ewan
    Commented Jan 11, 2019 at 14:37
  • one more question: does the Message Queue (ActiveMQ in this case) require a dedicated server? Or would it be running within midpoint? If so, how is it any different than just building a simple, memory-based system that sends and receives messages?
    – DemCodeLines
    Commented Jan 11, 2019 at 16:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.