1

Just like the title says, I want to use the pipeline design pattern for chaining related API calls. The context will store the data that will be used to create a payload. For example, I have the context object, a map of some sort, contain a "raw" data.

  1. First in the line is a formatter handler that formats the data in the context. I will store the result from the formatter in the context
  2. The next handler will use that data to build a request.
  3. The next handler will use the built request and send it.
  4. The next handler will parse the response from the request, store the required data.
  5. Go to #1 until the last request is sent.

The payload may use data generated/retrieved from other previous handlers, not just the one directly behind it. The list of handlers are loaded from a config file and are not fixed. The idea is for it to be generic and reusable. So let's say I have a RequestBuilder that will load a message "template" which will contain the headers and body of a request, apply the values on the template and send that as a request. This means that the id of the template should be in the context.

The problems I see are

  1. The context can get big and messy
  2. Handler n may need data from the "raw" data, from the data generated in the 1st request, etc, so there is a need of tracking

How can I implement this without getting messy? Should I implement it this way?

0

It looks like you fell into a trap of premature generalization. When you want to design generic solution you need to be able to analyze specific cases and then generalize if needed/possible. Otherwise you will end up with unusable solution, not good for most of the cases and a use will need to 'hack it' (it might be you).

Put yourself in a position of your solution client, this will give you good insights what is needed, what is practical, etc.

You should also consider what is already available on the market to not reinvent a wheel.
It is also good practice to think about maintenance of your solution - e.g. how violating Single Responsibility Principle affects future changes, what would be their impact.

0

I assume that by REST you mean web, hence asynchronous requests/responses. That's important because the pipeline should not progress until it has something to pass from pipe to pipe.

Handler n may need data from the "raw" data, from the data generated in the 1st request, etc, so there is a need of tracking

Contexts, messages, inputs, scopes whatever you pass from pipe to pipe should be data structures. They can bear whatever you need so, choose the one that allows you to do it. Try tho, not to use strict-typed structures. Use Maps or Tables to store and index content. Or Document Object Models.

The context can get big and messy

Then make it simple. Define clearly the SRP of each pipe. Give to each stage a single reason to change.

For example

  1. Pipe A calls service A, map the response to a Message and return the message.
  2. Pipe B transforms the resulting message. It creates and return a new and different message.
  3. Pipe C calls service B using the new message as input . It creates a new message from the result, adds a flag and return the message.
  4. Pipe D writes the new message into a file, checks the message looking for a flag. If the flag exists and is true, it sends an email. It returns a new message informing the sent email and the target.
  5. And so on...

A Pipe could be similiar to

public interface Pipe{
    void call (Message input, Pipeline pipeline);
}

public abstract class HttpCallPipe implements Pipe {

   private RestTemplate restTemplate;

   public HttpCallPipe (String url){
      this.restTemplate=new RestTemplate(url);
   }

   public void call(Message input, Pipeline pipeline){
      try {
        //Assuming that the http client works synchronously, 
        //otherwise we have to "await"
        Response response = restTemplate.doGet(input,Response.class);
        Message output = onSuccess(response, input);
        pipeline.next(output);
      }catch(Exception exception){
        onError(exception, input);
       //Maybe, we can continue 
       //wiht an alternative message
      }
   }       

   public abstract Message onSuccess(Response response, Message originalMessage);
   public abstract void onError(Throwable error, Message input);    
 }

Message can be similiar to

public interface Message {
    public Object get(String key);
}
public class MessageImpl implements Message {
    private final Map<String, Object> index;
    public MessageImpl(){...}
    public void set(String key, Objetc value){...}
    public Object get(String key){...}
}

The Pipeline is a chain of Pipes. Each pipe executes a command and tell the pipeline to continue with a new message.

The context can be a shared data structure within the pipeline, it's created by the Pipeline at the beginning, injected into the pipes and destroyed at the end of the pipeline. But I would dare to say that Message is all you need. Pipes are not state machines.

Finally,

  • For the sake of the scalability, make pipes stateless. Messages better if they are inmutables.

  • For the sake of the performance, destroy all the objects/functions at the end of the pipeline. Specially the Context if you have any. Don't bubble up events, like in Las Vegas, whatever happens in Pipe X, stays in Pipe X.


Note: The implementation will vary (a lot) from language to language. Functional and procedural programming languages are probably whom fit best for the matter. However, I have seen implementations in JavaScript doing this, and the source code was quite simple.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.