0

I am making a webapp that deals with money movement. All the financial actions are done through an API. For example, right now I can create an account for a user, add funds to their account, transfer funds to another account, etc. Currently I grab all the information on a user, their account balance, transactions etc. and store it on my local db.

My question is, is this the best approach? I am worried about some kind of data inconsistencies between what is in my db and what is in the source of truth (3rd party api).

One alternative would be to just query the data from the 3rd party any time it is requested and store next to nothing in my local db. Is there a name for this sort of problem?

1
  • A name? You could consider it a cache. Is it the best approach? Depends on what characteristics you want your app to have. I think you have to be more specific in what actual problem(s) you want to solve.
    – Rik D
    Commented Jun 24 at 16:56

1 Answer 1

3

This is an extremely common problem when building any sort of distributed system. This is such a foundational problem for service-oriented architecture and microservices that no single canonical term exists for this problem. That also means there is no single solution, either. Handling this "appropriately" involves a deep understanding of the business domain you are modeling and the technical constraints imposed on your system by those third party APIs. Additionally, your system might have its own domain-specific rules and technical constraints that you should be aware of before designing a solution.

From the information in your question, you appear to be dealing with finances and a web application. This provides a little guidance. The business domain is "finance" and "web application" helps provide technical constraints. You mention saving things to a local database, which a little ambiguous. A "local database" could mean a database which is deployed in your company's IT infrastructure or an IndexedDB in the browser. Either way, I think the problem remains largely the same.

I would recommend treating the third party API as the source of truth and keep minimal information in your system. Financial transactions are inherently dynamic, unpredictable, transient, and — even more frustratingly — reversible. Treat any information you save to your own database as a snapshot in time instead of an up-to-date repository of knowledge. This has some downstream implications for system design:

  1. You cannot guarantee ahead of time whether a transaction will succeed.
  2. Your system must treat the third party API as a source of truth for data, as well as the results of business operations.
    • You will need error handling code that communicates errors to the end user for cases where the user can resolve the problem.
    • And your will need error handling in place to gracefully record unresolvable problems in a way that allows your system to keep functioning and triggers business processes that allow people to manually correct issues.
  3. You cannot guarantee that a business operation is finished when the third party API responds back.
  4. You cannot guarantee your message was received by the third party API, and the third party API cannot guarantee their response was received by your system. The network between the two systems is inherently unreliable. This is known as the Two Generals' Problem, and is further illustrated by the fallacies of distributed computing.

If you do not properly handle #1, your application can suffer from time-of-check to time-of-use defects. Number 3 in the list is typically handled with eventual consistency whereby your system is programmed to treat business operations as an asynchronous process instead of a synchronous "command and get an authoritative response" process. Your system might call the third party API to start a business process which takes some time to finish. Your system might need to handle situations where the third party API reports back at some unknown point in the future whether the business process succeeded or failed.

While your question is a common problem, there is no easy answer. This is where the messiness of business processes runs head first into the technical quagmire of distributed computing. A combination of defensive programming, and business processes coupled with a deep understanding of the problem domain and technical constraints can help guide you to the right solution.

My best advice is to start by treating the third party API as the source of truth. Keep as little information in your system as necessary. Any transient information saved in your system should be treated as a snapshot in time that you assume is outdated the moment you receive it. You will need to build use cases into your application to handle all of the things that can go wrong, and when the end user cannot fix the situation, at least record enough information so people on your side can understand what went wrong and issue corrective actions to remedy the problem.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.