Java Design for Data enrichment based on logic defined in a database

Question

I have the following requirement.

Read data from a messaging Queue and process each message to enrich the message and then finally send the enriched message to a different system. The enrichment logic is in DB and it is based on message type and each logic can be of different language. Each message will have some type like, bank, insurance, ...

Sample enrichment logic data in Mongo DB,

{
  type : "Bank",
  logics : [
    {
      "key" : "location.phone",
      "type" : "javascript",
      "logic" : "function getPhone() { return '888-888-8888';}"
    },
    {
      "key" : "location.zip",
      "type" : "groovy",
      "logic" : "function getZip() { return '56781';}"
    },
    {
      "key" : "location.address",
      "type" : "groovy",
      "logic" : "function getAddress() { return '1234 new address';}"
    },
  ]
}

Now on receiving each message from the Queue, Check the type and enrich the data based on the logics for that type.

This is a spring boot based non-web Java Application.

My executing would start from the below run method.

public class AppRunner implements ApplicationRunner {

  // Other Bean injection is here


    @Override
    public void run(final ApplicationArguments args) throws Exception {
       // Application starts here ...
    }
}

I defined EnrichmentLogicDAO to load the logic from the DB and have a EnrichmentDBService for the DAO. Have an interface with a method exeucte( Command pattern ). There are 2 implementors. 1. Javascript, 2. Groovy

In Javascript executor,

ScriptEngine engine = new ScriptEngineManager().getEngineByName("JavaScript");

In Groovy executor,

ScriptEngine engine = new ScriptEngineManager().getEngineByName("groovy");

In both,

engine.eval(logic1);
engine.eval(logic2);
....

Load all the logic from DB. Javascript based logic should be loaded in JavaScriptExecutor engine and groovy based logic should be loaded in GroovyExecutor engine.

Question 1 :-

How to load all the logics to the corresponding engine?

Currently, I have my code to load the logic inside the AppRunner's run method and call the corresponding Executor class static method based on the type from DB to load the logics into the engine.

Question 2 :-

The code to execute the enrichment logic remain same for both the engine. Only the engine will be different based on the logic type.

I need some help on Loading the logics to the corresponding engine and invoke the logic on each message.

Also, I thought of using the Chain of responsibility pattern, so that the input message would be passed to the series of engines and the message will be enriched based on the logics loaded in each engine. Is there any other better approach.

Answer 1

To start things off simple, you should create a simple loop taking each "logic" and performing each operation from start to finish. So have add a dependency for your EnrichmentDBService and directly request the logics for your input.

Then for each logics, you apply the change. To apply each change, you should use an enum or some sort of direct mapping between the type "javascript" or "groovy" and the appropiate ScriptEngine instance pertaining to that particular language.

Using this mapping, grab the correct ScriptEngine instance and execute its code. What gets returned you must set to an object using "key" value presumably.

Once you've finished all logics in this way you're done! Return the finished object.

If order is not crucial and there are many logics, you could consider using a thread pool to handle the tasks. I wouldn't suggest Chain of Responsibility because it seems like the actual work being done is the same here. However, I fear that what they want from you is that you interpret the values returned as a sort of mask for you to apply to the actual data.

If, for instance, '888-888-8888' is a mask for applying a number, then you'd need some sort of baseline rule for this. Does this mean whenever you see '8', you represent this as a digit? What about '1234 new address'? Is '1234' supposed to be replaced with the street number? Why wouldn't you interpret this as a literal '1234'?

These questions you cannot answer yourself. The one providing the specifications should tell you this, and you need to look for potential inconsistencies and bring them to the attention of the one performing the analysis. Do this until all cases are covered and there are no inconsistencies before you begin coding.

Good luck!

Answer 2

Ok, let's do a draft top-down design using principles from object-orientation.

Read data from a messaging Queue and process each message to enrich the message and then finally send the enriched message to a different system.

That could mean something like this at the top:

@Override
public void run(final ApplicationArguments args) throws Exception {
   new Queue(...)
      .map(Message::enrich)
      .accept(EnrichedMessage::send)
      .start();
}

Now on receiving each message from the Queue, Check the type and enrich the data based on the logics for that type.

That sounds like the message would enrich itself based on type, so something like this:

public final class Message {
    private final Enrichment type;
    private final Data data;
    ...
    public EnrichedMessage enrich() {
        Data enrichedData = type.enrich(data);
        return new EnrichedMessage(enrichedData);
    }
}

public interface Enrichment {
    void enrich(Data data);
}

So at this point we only have to implement Enrichments (and the plumbing for queue and database, but that is not that exciting :).

We probably need one for each language.

public final class JavascriptEnrichment implements Enrichment {
    ...
    public JavascriptEnrichment(String script) {
        ...
    }
    ...
}

and so on. Then one for a list of enrichments.

public final class Enrichments implements Enrichment {
    private final List<Enrichment> enrichments;
    ...
    @Override
    public Data enrich(Data data) {
        ...enrich with all...
    }
}

Then of course we need one Enrichment that makes the lookup from Mongo based on type. This will be used by the Queue on deserialization to put into the Message.

public final class TypeEnrichment implements Enrichment {
    private final String type;
    ...
    @Override
    public Data enrich(Data data) {
        ...load "Enrichments" from mongo based on type...
    }
}

So this is a sketch of an overall design that I would expect based on your requirements. HTH