I need to build a data store for transactions that meets the following requirements:
- A 'transaction' is effectively a state machine which moves through a number of statuses during its lifecycle and the incoming data messages represent changes to a transaction's state (eg, transaction moves from 'Created' to 'Open' to 'Billed')
- Incoming data is either pulled from third party APIs, or received though calls to the system's own public API.
- Each incoming data message is in some proprietary format which needs to be transformed into a single consistent format and stored
- The consistent view of the transaction data is then published to clients in two ways:
- Via a static REST endpoint
- Via a real-time feed of change events
The system needs to be resilient and fault tolerant - dropping events or changes to a transaction is forbidden.
My thoughts are:
- Have a very simple input layer to drop incoming messages onto a queue
- An adapter layer would read from the queue to parse, transform and store the incoming messages
- The static API is relatively trivial
- I am not sure how to design the real-time aspect.
Some questions / design considerations:
- Should the adapter layer publish events after storing the data, or is this mixing concerns?
- Should I store the transaction changes as an event stream and materialise a view for any queries?
- Can anyone point me to some reference architecture for similar systems?