Designing for many third-party REST API integrations

Question

Not sure if this is the right forum for this, but I've come across a need for this pattern a few times lately, and I would love to get some opinions on options for how to handle it.

Say I have an app that takes data from an arbitrary number of third-party applications and wants to parse it into a unifying format:

{
    "field_1": "value_1",
    "field_2": "value_2"
}

So I might have integration 1 return a response that looks like

{
    "different_field_1_name": "desired_field_1_value",
    "field_2": "desired_field_2_value"
}

And integration 2 return a response that looks like

{
    "different_field_1_name_again": "desired_field_1_value",
    "different_field_2_name": "desired_field_2_value",
    "unneeded_field": "unneeded_value"
}

etc.

Ideally, adding new integrations should be as painless as possible. But every time I've had to do something like this, things like authentication, response formats, weird API quirks, etc. always make it feel like I'm starting from scratch with new integrations. Does anybody have examples of open-source applications that I can look at that do this sort of thing well? Or is there an obvious pattern I'm missing here that makes this easier?

Answer 1

Have a look at the adapter pattern (I'm guessing probably already know it). It is basically just a glorified mapper function:

DomainObject toDomain(ThirdPartyObject object) { ... } 

What you then need to do is just fetch the third party data. How that looks really depends on the provider and I don't think you can abstract it more than just:

List<ThirdPartyObject> fetch() { ... }

so basically you can create:

public abstract class IntegrationAdapter<T> {
  publict List<DomainObject> adapt() {
    List<T> thirdPartyObjects = fetch();
    return thirdPartyObjects.stream.map(this::toDomain).toList()
  }

  private abstract List<T> fetch();
  private abstract DomainObject toDomain(T object);
}

You can then make implementations of this abstract class.

Or to have a more modular and easier to test components you could do:

public interface Client<T> {
  List<T> fetch();
}

public interface Mapper<D, T> {
  D toDomain(T object);
}

public interface DomainObjectRepository {
  List<DomainObject> getDomainObjects();
}

class ThirdPartyClient implements Client<ThirdPartyObject> { ... }

class DoTpoMapper implements Mapper<DomainObject, ThirdPartyObject> { ... }

class ThirdPartyRepository implements DomainObjectRepository {
  public ThirdPartyRepository(ThirdPartyClient, DoTpoMapper) {...}

  List<DomainObject> getDomainObjects() {
    List<T> thirdPartyObjects = this.thirdPartyClientfetch();
    return thirdPartyObjects.stream.map(doTpoMapper::toDomain).toList();
  }
}

You can then write integration and unit tests for the client, the repository and the mapper, which you would write just once in a generic fashion, and could then reuse them when creating new adapters.