-4

I work for this Accounting company - as a Product Owner - that uses several external services (one for Contract management, another one for Accounting Demands management, a third one for Financial purposes and so on).

My IT Director wants us to build an Event-Driven Architecture to gather all the events that happen externally and bring them into our NoSQL in as-close-as-possible realtime.

The problem specifically is: how can I gather this data from external sources? What would be the best procedure for hundreds of events every day from different services? Consuming their APIs periodically?

Can you guys shine a light on how this can be properly done?

Improve this question
2
  • 1
    Sharing your research helps everyone. Tell us what you've tried and why it didn't meet your needs. This demonstrates that you've taken the time to try to help yourself, it saves us from reiterating obvious answers, and most of all it helps you get a more specific and relevant answer. Also see How to Ask
    – gnat
    Aug 31, 2020 at 14:08
  • What exactly are these "services" from a technical point of view? Web services? With some REST Api? Do they provide some event mechanism on their own? Or is your problem that they don't?
    – Doc Brown
    Aug 31, 2020 at 18:54

1 Answer 1

Reset to default
2

First, building a Event Driven Architecture doesn't necessarily mean that your application interacts with the external world in an event driven way. Most of 3rd party services provide REST (HTTP) APIs that lead to synchronous communication patterns. Setting up some workaround that involves polling could help but there are two pitfalls I would consider

  • If I was a 3rd party provider I wouldn't be very happy that you call me every second. Decent API implementations have guards against this kind of usage (throttling, rate limiting)

  • You could resolve the previous point setting a long polling period, but this would make your fake-event-driven interaction useless because you cannot leverage its bigger benefit: timely status updates

So, the first step here is to check what interaction pattern your providers allow. Notice that "event driven API" doesn't necessarly mean that they will send you events through message queues, even webhooks could work well to bridge your event driven app with the outside

If there is no way to manage it, consider the idea to build an "anticorruption layer". some component can decide to fetch external data whenever a significant internal event occours, compare the outcome with the last recorded state and decide to advertise a state change in internal representation of the external resource.

In any case keep in mind that working with the outside in an event-driven way means to change the way your application performs distributed transaction, switching to the famous "Saga Pattern". I won't go deep on this because it's out of scope, but the basic idea there is that a distributed transaction can have a progress status. A component will keep that progress status up to date listening events and knows when and how to recover from partial failures

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.