A Design Pattern for a component which both writes to and reads from an ordered event log

Question

I am searching for a good design for a set of components I am writing for a system.

I believe it is very likely there is a Design Pattern, or set of Design Patterns, which could be combined to solve this problem in a standardized way.

There are several components which write to an ordered event log. Two of them are:

• ClientDatabase: A data structure which holds information in memory about clients
• ClientOrderDatabase: A data structure which holds information in memory about orders made by clients

The problem I have is that these components both read to and write from the log. There is no obvious dependency relationship between them. My instinct tells me that this is a solved problem, and that there is probably a design pattern, or set of design patterns which can be used to build a good archetecture to solve this problem. I will explain the details a bit later.

To explain what the purpose of the "ordered event log" is:

• The ordered event log is being used in place of a database or other data store
• Mutating operations made to the ClientDatabase and ClientOrderDatabase are recorded to the event log, with sequential "in time" ordering
• The purpose of this log is for data persistence. If the system restarts, it can read the sequence of events written to the log to restore its state
• This is actually how databases work. The front end of a database consists of an ordered event log with transactional capabilities built in. The actual data in a database is maintained by reading the log and updating the internal state

If you are wondering why I am building this rather than using a database, this is a demonstration project. The purpose is just to demonstrate some computer science concepts.

Design Patterns

I think it will be easiest to understand what this system does by considering the flow of some message through the system.

Here is an example which describes how a new client creates an account:

• A REST API endpoint /create_user is called with a data payload containing a username and password
• ClientDatabase.create_user(username, password) is called
• This will create a new user (add data to a dictionary) if the username does not already exist, or throw an exception if the username has already been registered
• To persist data, the ClientDatabase needs to write a message to the event log. This only happens if the user is created successfully
• The ClientDatabase should be testable, so it probably should not hold a reference to the event log
• However, it might be a good design solution for a decorator to hold a reference to the event log as well as an instance of a ClientDatabase

class ClientDatabaseLogged():

    def create_user(self, username, password):
        try:
            self._client_database.create_user(username, password)        
        self._event_log.write_user_created(username, password)


class EventLog():

    def write_user_created(self, username, password):
        # self._event_log = open('event_log.txt', 'w')
        self._event_log.write(f'CREATE_USER {username} {password}')
        self._event_log.flush()

However, this design has created a problem: What should the system startup logic be?

• When the system starts, it should initialize an instance of the ClientDatabase. Initially it will be empty and contain no data
• To restore the persisted system state, the system should read the event log and re-process these events
• The event log talks about multiple things. It contains a log history of instructions which should be dispatched to various system components to initialize them
• For example:
• CREATE_USER example_user password123
• This is an instruction which should be sent to the ClientDatabase
• CREATE_ORDER example_user example_product_code example_quantity
• This is an instruction which should be sent to the ClientOrderDatabase

I am not sure what a good design would be here.

• The problem seems to be that if I write a ClientDatabaseLogged which holds a reference to an EventLog, there is no seemingly sensible way to drive the startup process
• It is obvious that, as part of the startup process, the EventLog file must be read line by line and parsed. Each line in the event log is an instruction
• The type of instruction seen determines the destination component
• I could write an interface EventLog.poll_for_instruction(), but it isn't obvious how this should behave under normal operation when the event log is being written to rather than read from during the start up process
• It seems to me that a more sensible design would be for the EventLog.__init__ (constructor) function to read the event log file line by line if it exists, and send these instructions to the correct consumer component
• This creates a circular dependency however. The EventLog would need to hold a reference to each consumer component to send data to them during the startup process, and the components would also need to hold a reference to the EventLog to send data to it under normal running mode when the components act as producers rather than consumers

If this question isn't totally clear I will try and provide more clear and detailed information if neccessary.

Answer 1

There were some upvotes on this recently so I thought I would add some information about how I actually solved this problem in the end.

• I decided that an EventLogger should not be an adapter class which wraps some inner implementation
• Instead, I built the EventLog class as its own separate entity, independent from the business logic. I wrote it such that it handled the IO for abstract message types, and defined the concrete implementations of those elsewhere.

I then wrote a third type which connected both of these together.

class NameNotImportant:
    # __init__
    self.event_log = EventLog(...)
    self.business_logic = BusinessLogic(...)

Sorry, I don't have time to add much detail here, but this was the general idea.

The problem with writing an event log as an adapter which does all the IO is that it strongly coupled together the direction of the dependency of the BusinessLogic and the EventLogger. This works well when processing events, but does not work well when starting up and reading events from a file because you have a weird asymmetry.