22 votes
Accepted

Differences between API Gateways and ESBs?

The reason you're getting the concepts jumbled up is that the vendors are selling them in a package. But they are definitely separate concepts. An API Gateway provides a central access point for ...
user avatar
13 votes

How do you scale your integration testing?

I worked at a place that took 5 hours (across 30 machines) to run integration tests. I refactored the codebase and did unit tests instead for the new stuff. The unit tests took 30 seconds (across 1 ...
user avatar
  • 107k
9 votes

How do you scale your integration testing?

Integration tests will always be long running as they should mimic a real user. For this very reason you shouldn't run them all synchronously! Given that you are already running stuff in the cloud it ...
user avatar
  • 64.1k
9 votes
Accepted

How to handle database errors after a successful 3rd party payments API response?

Trying to cover failures like this with business logic and retries is in my view a mistake. You can always think up a scenario which isn't covered by your logic. In your case, case the DB write has ...
user avatar
  • 64.1k
8 votes

Reasons NOT to use email for system integration

How do I nicely explain to a customer that if the message is really critical and time sensitive then email (SMTP) is NOT the way to do it. By knowing why it's not. Name one real need that the email ...
user avatar
7 votes
Accepted

What is a "streaming app", exactly?

A streaming app is an app that consumes a stream of data. A stream of data is transmitted data formatted in a way that can be useful even when incomplete. Since partial stream data does not require ...
user avatar
7 votes
Accepted

How to "state of the art" integrate multiple sub-modules with duplicate dependencies?

Your incompatible Logger implementations cannot be linked because they have the same name. This leads to a simple solution that will solve all your problems: Incompatible versions should have ...
user avatar
  • 123k
6 votes
Accepted

Real time data synchronization techniques between two systems

First, as the comments point out, you aren't really talking about realtime here -- that has a pretty strict definition and implies lots of stuff that isn't really on the table. You are talking about ...
user avatar
6 votes
Accepted

How to deal with bad third party APIs in a microservices architecture?

I encounter this type of thing all the time, there are some unbelievable commercial SOAP/XML APIs out there. What I normally do is basically two things: Define a facade for the functionality my ...
user avatar
4 votes
Accepted

Data integration from heterogeneous sources

The fundamental problem here is that REST is not a standard. It is an architectural style for web APIs. While there are mechanisms to make these APIs self-describing, and there are web-service ...
user avatar
  • 123k
4 votes
Accepted

Good idea to use source control with my Lambda functions?

As a rule you always want to have version control for your deployed code - typically this means a source code repository separate from your application. In the case of AWS Lambda, there is a built in ...
user avatar
  • 7,795
3 votes

What is needed write an interface to Microsoft Dynamics GP web services?

I was trolling around looking for examples of programmers shelling GP Web Services with Java and came across this article. As responses are a couple years old this is more informational for the next ...
user avatar
3 votes

What is needed write an interface to Microsoft Dynamics GP web services?

The preferred approach towards customizing the Great Plains (GP) is to access data via database. You can write a application in any programming language you are comfortable with which can connect to ...
user avatar
3 votes

Points to Understand Architecture of an Application/System

If you built a house and you want to describe it to someone, would you talk about what kind of flooring material you used? About what kind of hammer and tools you used? About the plumbing? You might, ...
user avatar
3 votes

How do you scale your integration testing?

Sounds like you have way too many integration tests. Recall Test pyramid. Integration tests belong in the middle. As an example take a repository with method set(key,object), get(key). This ...
user avatar
3 votes
Accepted

How to design error monitoring for batch jobs with multiple and diverse steps?

The problem is the design of alerting that the integration of that specific unit failed. Take a step back from the technical solution and think about it from a business perspective first. What types ...
user avatar
  • 6,872
3 votes

Reasons NOT to use email for system integration

I agree with everything CandiedOrange mentioned (including the how to explain x to y comment). It's all about the clients needs. I've built systems that rely on smtp email. It works fine and it suited ...
user avatar
  • 443
3 votes

Is there a better way of communicating between systems?

Apache Kafka works in realtime, and is somehow in vogue for solving the subsystem communication problems that used to be solved with centralized databases or queue management systems. As to the XML, ...
user avatar
  • 2,273
3 votes

How can I inform my team when components are ready for integration?

If you don't want to add a new tool then use what you have. I like that task descriptions are editable in most ticketing tools. Task: 54328 - Make "Number of Dependents" spinner stop allowing ...
user avatar
3 votes
Accepted

Legacy modernisation - Parallel systems vs Extending Original application?

I'm assuming you have a small team and a limited budget, so my answer assumes the following is true: Maintaining 2 separate applications would be a big strain on the team The work to integrate the ...
user avatar
3 votes

Bidirectional sync between two databases with differing schemas

You should not do this. Doing a unidirectional sync is challenging, adds a pile of constraints to your system, but is a fairly well known problem at this point. When you make things bidirectional, ...
user avatar
  • 107k
3 votes
Accepted

Sending JSON in body of an email

This is a slightly creative architecture, but generally fine. Some caveats: Email is notoriously unreliable: emails may arrive delayed, out of order, or not at all. Emails are not necessarily ...
user avatar
  • 123k
3 votes

Sending JSON in body of an email

JSON in the email body is problematic. Mostly because it’s between hard and impossible to guarantee that an email body isn’t modified by anyone. JSON in an attachment is just fine, especially if you ...
user avatar
  • 37.4k
3 votes

What happens after the ETL process?

An ETL is just a procedure. You: Extract the data from any source. In your case, this would be the CSV files, but it could be anything: data from an external service, data stored in some proprietary ...
user avatar
3 votes
Accepted

Is it reasonable to pass file URLs to microservices instead of the file contents?

I'd say number 2 is perfectly fine in theory but you can improve on it, which would alleviate some of the concerns that are leading you to doubt if it's a valid approach. In simpler terms, your cookie ...
user avatar
  • 37.3k
2 votes

How do you scale your integration testing?

In my experience in an Agile or DevOps environment where continuous delivery pipelines are common, integration testing should be carried out as each module is completed or adjusted. For example, in ...
user avatar
  • 31
2 votes
Accepted

Simple messaging pattern to replace shared DB?

Lets to do an exercice of creativity A's API Rest as webservice: In order to feed B and unbind B from A's data model , we need an interface that will act like a contract. This contract is going to ...
user avatar
  • 11.6k
2 votes

How can I inform my team when components are ready for integration?

A board filled with sticky notes (or a Jira board) is as simple as you can get. A glimpse on the board should give enough information about the progress of the team, and if it doesn't, it means that ...
user avatar
1 vote

Reasons NOT to use email for system integration

You don't! You just make sure the client is aware of the distinctions: Email is more inherently optimistic, and it's more inherently asynchronous. When the client sends a message, it assumes the ...
user avatar
  • 13.2k

Only top scored, non community-wiki answers of a minimum length are eligible