Skip to main content
Share Your Experience: Take the 2024 Developer Survey
22 votes
Accepted

How to handle backpressure in message queue

A good analogy here is to think of a dam on a river. The river corresponds to the incoming data, and the dam to your consumer. There are three possibilities at any given point: The incoming river's ...
JimmyJames's user avatar
  • 27.5k
9 votes

Which is the best option to exchange large amount of data in a micro-service architecture?

There are some important aspects you should consider first. Streaming Let's imagine the 100 MB file is received by the service A which transfers it to service B, which, in turn, uses service C to do ...
Arseni Mourzenko's user avatar
8 votes
Accepted

Does AWS Kinesis guarantee a specific type of ordering?

The term that I would use is "arbitrary" rather than "random". But your observation is correct: the sequence numbers merely indicate the sequence in which records were added to the partition. They are ...
kdgregory's user avatar
  • 5,250
8 votes
Accepted

Invoking lambda directly from S3 vs going via SQS

I think the most robust approach is S3 -> SNS -> SQS -> Lambda SNS gives you that pub sub endpoint so you can attach more things to the event if required. With SNS you can publish a ...
Ewan's user avatar
  • 76.4k
6 votes
Accepted

NoSQL or SQL for my situation?

We don't know what you're going to persist in the db (vs. deal with in memory), what write volume, what is in transactions, what read volume, what joins etc.. Further, we don't know about evolution ...
Erik Eidt's user avatar
  • 34.1k
6 votes
Accepted

Server architecture for short bursts of ~150 parallel CPU-heavy subqueries

Since you are CPU-limited, you need to get your hands on 150 CPU cores, one for each thread. This rules out a single server, since a server of such proportions would be prohibitively expensive – and ...
amon's user avatar
  • 134k
6 votes

How to organize projects in AWS?

AWS Organizations is the standard approach to managing a hierarchy of accounts. However, it does not allow billing to be consolidated at arbitrary points in the organization; it's a tree, not a graph. ...
guest's user avatar
  • 131
4 votes
Accepted

DB migration strategy for docker containers in AWS ECS

A change should only change that which is necessary. Removing unused columns in a DB schema isn't a necessary change, it's a non-functional, maintainability kind of change. So there's no reason to ...
Paul's user avatar
  • 3,317
4 votes

Watermark on Video implementation , Server or Client

There are two considerations here: Adding a watermark takes quite a bit of processing power, especially for videos. It might be possible that a phone is capable of adding the watermark, but it will ...
Bart van Ingen Schenau's user avatar
4 votes
Accepted

What's stopping Cloud Platforms from owning the whole deployment process?

Taking your example of AWS, it can indeed manage the entire deployment process, including deploying the full environment from Cloud Formation config files through to the source code of the application....
David Arno's user avatar
  • 39.3k
4 votes

Fire aws lambda at a specific time

I had to tackle a similar problem and depending on the volume of scheduling that needs to happen, an elegant solution would be to use AWS Step Function with a Wait Step, see https://docs.aws.amazon....
Simon's user avatar
  • 41
4 votes
Accepted

Re-architecting CPU intensive Node application to handle multiple users

Nodejs is good for IO intensive tasks but may not be good for CPU intensive tasks. The reason is that it runs on the event loop, which runs on a single thread. There may be several possible ...
lennon310's user avatar
  • 3,212
4 votes

How to handle backpressure in message queue

How throttling helps to solve backpressure problem? If the TPS is consistent, Wouldn't it create the same problem even when producer retry after delay? At the end some producers will exhaust retries ...
Useless's user avatar
  • 12.6k
4 votes

Do Lambdas(AWS) stop making sense after a certain number of functions?

Working with Lambdas really requires some deployment automation tooling to manage all of these things for you. At my last job, we used Serverless Framework, but after a year I was starting to feel ...
Scott Simontis's user avatar
4 votes
Accepted

Quiz Site Software Architecture Question

Would these constant queries slow down my application? How much? I've learned the hard way over many years to measure a performance problem first before trying to solve it. Even if you measure a ...
Greg Burghardt's user avatar
4 votes

Creating associated resources before creating the main one in a REST setting

Perhaps you're documenting a visit to the radiologist, so in addition to pulse rate and blood pressure we're recording lots of giant X-ray images in lots of S3 image files. S3 has no efficient bulk ...
J_H's user avatar
  • 7,156
3 votes

Send multiple record types to a single Kinesis stream or not?

It depends on how your consumers process the streams. Every consumer processes each type of event, independently This might be a typical logging stream: you care that the events happen, but aren't ...
kdgregory's user avatar
  • 5,250
3 votes
Accepted

Messaging between microservices on AWS

"We are considering creating a queue for each microservice and publishing the messages to the queues via SNS." Yes, then your queues can subscribe to the notification and you no longer need the ...
Ewan's user avatar
  • 76.4k
3 votes

Amazon RDS read replica without support from application

When designing databases you have two basic choices: Scale-Up or Scale-Out. In the first case you use bigger machines (instances), for the second case you add more machines (instances) and this means ...
John Hanley's user avatar
3 votes

Can a Database Scale Up?

Of course you can scale databases. Also, with a strict interpretation of monolith and microservice there generally shouldnt be this kind of performance scaling advantage to microservices. A monolith ...
Ewan's user avatar
  • 76.4k
3 votes

Are AWS Lambda functions a good fit for the use case of fetching huge amount of records from database on User request?

I believe there is a 6Mb limit on the request/response, which would make it unsuitable. https://docs.aws.amazon.com/lambda/latest/dg/limits.html Ideally you want a resumable stream of data for large ...
Ewan's user avatar
  • 76.4k
3 votes
Accepted

What is the use of ENI in AWS?

ENI is a network interface, which embodies some configuration, which can include IP addresses, an elastic IP (EIP), various security groups (firewall rules), a MAC address, a source/destination check ...
Rob's user avatar
  • 164
3 votes
Accepted

Scheduled AWS Lambda to Update Database via API Call

You should probably look over this: AWS lambda best practices. The first bullet is one thing you should definitely consider: Initialize SDK clients and database connections outside of the ...
JimmyJames's user avatar
  • 27.5k
3 votes
Accepted

Why should I use containers instead of deploying build artifacts directly to Azure App Services or AWS Elastic Beanstalk?

On a very high level, you are not missing out on anything crucial. Docker (and Kubernetes) provide a runtime environment, control plane, deployment, network abstraction etc. which are also provided by ...
Martin K's user avatar
  • 2,917
3 votes

Why should I use containers instead of deploying build artifacts directly to Azure App Services or AWS Elastic Beanstalk?

A full answer to your question depends on the complexity of deployment. If you are looking to host a single ASP.Net (core or classic) website, with no additional services, then I would recommend the ...
Berin Loritsch's user avatar
3 votes

AWS Architecture: How to search in Dynamodb?

Which AWS tool should I use for searching the DynamoDB? You don't. DynamoDB is designed for quick and efficient lookup by key and by key only1. If you are attempting to search DynamoDB as any part of ...
Philip Kendall's user avatar
3 votes
Accepted

Foreign Keys and Eventual Consistency

So the foreign keys come from a source system (maybe an OLTP database) where they are immutable, and the metrics database (a kind of analytics DB) reuses them. For such a situation, I would say it is ...
Doc Brown's user avatar
  • 208k
3 votes

In Publish-subscribe pattern can a microservice publish message for itself

We did this in one of our services. It felt cleaner because everything was going through the same interface instead of having a separate 'internal-to-the-service' interface that behaved slightly ...
Karl Bielefeldt's user avatar
3 votes
Accepted

Multiple Lambdas, one database?

Recommendations like “every service should have its own database” are just guidelines. The point of this guideline is that if a database were to be shared between multiple services, this would obscure ...
amon's user avatar
  • 134k
3 votes

Using Fargate (serverless compute) to replace leadership election

Using Fargate is saying, I don't want to deal with the instance. This includes things like instance patching, etc. Those things are done by cloud provider. In some cases, that is perfectly fine. ...
Jon Raynor's user avatar
  • 11.5k

Only top scored, non community-wiki answers of a minimum length are eligible