22
votes
Accepted
How to handle backpressure in message queue
A good analogy here is to think of a dam on a river. The river corresponds to the incoming data, and the dam to your consumer. There are three possibilities at any given point:
The incoming river's ...
- 28.5k
10
votes
Which is the best option to exchange large amount of data in a micro-service architecture?
There are some important aspects you should consider first.
Streaming
Let's imagine the 100 MB file is received by the service A which transfers it to service B, which, in turn, uses service C to do ...
- 136k
8
votes
Accepted
Does AWS Kinesis guarantee a specific type of ordering?
The term that I would use is "arbitrary" rather than "random". But your observation is correct: the sequence numbers merely indicate the sequence in which records were added to the partition. They are ...
- 5,270
8
votes
Accepted
Invoking lambda directly from S3 vs going via SQS
I think the most robust approach is
S3 -> SNS -> SQS -> Lambda
SNS gives you that pub sub endpoint so you can attach more things to the event if required.
With SNS you can publish a ...
- 78.6k
6
votes
Accepted
Server architecture for short bursts of ~150 parallel CPU-heavy subqueries
Since you are CPU-limited, you need to get your hands on 150 CPU cores, one for each thread. This rules out a single server, since a server of such proportions would be prohibitively expensive – and ...
- 135k
6
votes
Accepted
NoSQL or SQL for my situation?
We don't know what you're going to persist in the db (vs. deal with in memory), what write volume, what is in transactions, what read volume, what joins etc..
Further, we don't know about evolution ...
- 34.3k
6
votes
How to organize projects in AWS?
AWS Organizations is the standard approach to managing a hierarchy of accounts. However, it does not allow billing to be consolidated at arbitrary points in the organization; it's a tree, not a graph.
...
- 131
4
votes
Accepted
DB migration strategy for docker containers in AWS ECS
A change should only change that which is necessary. Removing unused columns in a DB schema isn't a necessary change, it's a non-functional, maintainability kind of change. So there's no reason to ...
- 3,317
4
votes
Watermark on Video implementation , Server or Client
There are two considerations here:
Adding a watermark takes quite a bit of processing power, especially for videos. It might be possible that a phone is capable of adding the watermark, but it will ...
- 75.7k
4
votes
Accepted
What's stopping Cloud Platforms from owning the whole deployment process?
Taking your example of AWS, it can indeed manage the entire deployment process, including deploying the full environment from Cloud Formation config files through to the source code of the application....
- 39.4k
4
votes
Fire aws lambda at a specific time
I had to tackle a similar problem and depending on the volume of scheduling that needs to happen, an elegant solution would be to use AWS Step Function with a Wait Step, see https://docs.aws.amazon....
- 41
4
votes
Accepted
Re-architecting CPU intensive Node application to handle multiple users
Nodejs is good for IO intensive tasks but may not be good for CPU intensive tasks. The reason is that it runs on the event loop, which runs on a single thread.
There may be several possible ...
- 3,242
4
votes
How to handle backpressure in message queue
How throttling helps to solve backpressure problem? If the TPS is consistent, Wouldn't it create the same problem even when producer retry after delay? At the end some producers will exhaust retries ...
- 12.7k
4
votes
Do Lambdas(AWS) stop making sense after a certain number of functions?
Working with Lambdas really requires some deployment automation tooling to manage all of these things for you. At my last job, we used Serverless Framework, but after a year I was starting to feel ...
- 141
4
votes
Accepted
Quiz Site Software Architecture Question
Would these constant queries slow down my application? How much?
I've learned the hard way over many years to measure a performance problem first before trying to solve it. Even if you measure a ...
- 40.8k
4
votes
Creating associated resources before creating the main one in a REST setting
Perhaps you're documenting a visit to the radiologist,
so in addition to pulse rate and blood pressure we're
recording lots of giant X-ray images in lots of S3 image files.
S3 has no efficient bulk ...
- 7,620
4
votes
Is it worth it to use a NoSQL database only for high availability and fast reading, while already using Postgres?
Choosing a technology to solve a problem before the problem is well understood is a good way for wasting time and energy.
First of all, the volumes that you indicate are not very high. I have seen a ...
- 79.9k
3
votes
Amazon RDS read replica without support from application
When designing databases you have two basic choices: Scale-Up or Scale-Out. In the first case you use bigger machines (instances), for the second case you add more machines (instances) and this means ...
- 131
3
votes
Send multiple record types to a single Kinesis stream or not?
It depends on how your consumers process the streams.
Every consumer processes each type of event, independently
This might be a typical logging stream: you care that the events happen, but aren't ...
- 5,270
3
votes
Accepted
Messaging between microservices on AWS
"We are considering creating a queue for each microservice and publishing the messages to the queues via SNS."
Yes, then your queues can subscribe to the notification and you no longer need the ...
- 78.6k
3
votes
Can a Database Scale Up?
Of course you can scale databases.
Also, with a strict interpretation of monolith and microservice there generally shouldnt be this kind of performance scaling advantage to microservices. A monolith ...
- 78.6k
3
votes
Are AWS Lambda functions a good fit for the use case of fetching huge amount of records from database on User request?
I believe there is a 6Mb limit on the request/response, which would make it unsuitable.
https://docs.aws.amazon.com/lambda/latest/dg/limits.html
Ideally you want a resumable stream of data for large ...
- 78.6k
3
votes
Accepted
What is the use of ENI in AWS?
ENI is a network interface, which embodies some configuration, which can include IP addresses, an elastic IP (EIP), various security groups (firewall rules), a MAC address, a source/destination check ...
- 164
3
votes
Accepted
Scheduled AWS Lambda to Update Database via API Call
You should probably look over this: AWS lambda best practices. The first bullet is one thing you should definitely consider:
Initialize SDK clients and database connections outside of the ...
- 28.5k
3
votes
Accepted
Why should I use containers instead of deploying build artifacts directly to Azure App Services or AWS Elastic Beanstalk?
On a very high level, you are not missing out on anything crucial. Docker (and Kubernetes) provide a runtime environment, control plane, deployment, network abstraction etc. which are also provided by ...
- 2,917
3
votes
Why should I use containers instead of deploying build artifacts directly to Azure App Services or AWS Elastic Beanstalk?
A full answer to your question depends on the complexity of deployment. If you are looking to host a single ASP.Net (core or classic) website, with no additional services, then I would recommend the ...
- 46.2k
3
votes
AWS Architecture: How to search in Dynamodb?
Which AWS tool should I use for searching the DynamoDB?
You don't. DynamoDB is designed for quick and efficient lookup by key and by key only1. If you are attempting to search DynamoDB as any part of ...
- 25k
3
votes
Accepted
Foreign Keys and Eventual Consistency
So the foreign keys come from a source system (maybe an OLTP database) where they are immutable, and the metrics database (a kind of analytics DB) reuses them.
For such a situation, I would say it is ...
- 212k
3
votes
In Publish-subscribe pattern can a microservice publish message for itself
We did this in one of our services. It felt cleaner because everything was going through the same interface instead of having a separate 'internal-to-the-service' interface that behaved slightly ...
- 148k
3
votes
Accepted
Multiple Lambdas, one database?
Recommendations like “every service should have its own database” are just guidelines.
The point of this guideline is that if a database were to be shared between multiple services, this would obscure ...
- 135k
Only top scored, non community-wiki answers of a minimum length are eligible