Backend architecture for a datastore with real-time events?

Question

I need to build a data store for transactions that meets the following requirements:

• A 'transaction' is effectively a state machine which moves through a number of statuses during its lifecycle and the incoming data messages represent changes to a transaction's state (eg, transaction moves from 'Created' to 'Open' to 'Billed')
• Incoming data is either pulled from third party APIs, or received though calls to the system's own public API.
• Each incoming data message is in some proprietary format which needs to be transformed into a single consistent format and stored
• The consistent view of the transaction data is then published to clients in two ways:
• 1. Via a static REST endpoint
• 2. Via a real-time feed of change events

The system needs to be resilient and fault tolerant - dropping events or changes to a transaction is forbidden.

My thoughts are:

• Have a very simple input layer to drop incoming messages onto a queue
• An adapter layer would read from the queue to parse, transform and store the incoming messages
• The static API is relatively trivial
• I am not sure how to design the real-time aspect.

Some questions / design considerations:

• Should the adapter layer publish events after storing the data, or is this mixing concerns?
• Should I store the transaction changes as an event stream and materialise a view for any queries?
• Can anyone point me to some reference architecture for similar systems?

Answer 1

Ok first of all you need to define what is the real time requirement for your system, or what latency has to satisfy to be considered real time.

For example: 1. you get a request for a transaction

a. check that transaction is valid and fully defined

b. appropriately respond to the transaction in the process verify that the transaction is doable

All of that has to happen in a quantum of time that you consider valid for your real time lets say 10 ms. The question rises what happens after 10 ms do you say resend transaction or bad request?

Next problem with such things is you need to profile everything and benchmark. This will give you the places where things are not going according to plan. Than repaire and profile again.

I hope this gives you a plan.