22
votes
Accepted
How to handle backpressure in message queue
A good analogy here is to think of a dam on a river. The river corresponds to the incoming data, and the dam to your consumer. There are three possibilities at any given point:
The incoming river's ...
- 25.2k
9
votes
Which is the best option to exchange large amount of data in a micro-service architecture?
There are some important aspects you should consider first.
Streaming
Let's imagine the 100 MB file is received by the service A which transfers it to service B, which, in turn, uses service C to do ...
- 135k
8
votes
Accepted
Does AWS Kinesis guarantee a specific type of ordering?
The term that I would use is "arbitrary" rather than "random". But your observation is correct: the sequence numbers merely indicate the sequence in which records were added to the partition. They are ...
- 5,220
7
votes
Accepted
Invoking lambda directly from S3 vs going via SQS
I think the most robust approach is
S3 -> SNS -> SQS -> Lambda
SNS gives you that pub sub endpoint so you can attach more things to the event if required.
With SNS you can publish a ...
- 71k
6
votes
Accepted
NoSQL or SQL for my situation?
We don't know what you're going to persist in the db (vs. deal with in memory), what write volume, what is in transactions, what read volume, what joins etc..
Further, we don't know about evolution ...
- 33.4k
6
votes
Accepted
Server architecture for short bursts of ~150 parallel CPU-heavy subqueries
Since you are CPU-limited, you need to get your hands on 150 CPU cores, one for each thread. This rules out a single server, since a server of such proportions would be prohibitively expensive – and ...
- 133k
6
votes
How to organize projects in AWS?
AWS Organizations is the standard approach to managing a hierarchy of accounts. However, it does not allow billing to be consolidated at arbitrary points in the organization; it's a tree, not a graph.
...
- 131
4
votes
Accepted
DB migration strategy for docker containers in AWS ECS
A change should only change that which is necessary. Removing unused columns in a DB schema isn't a necessary change, it's a non-functional, maintainability kind of change. So there's no reason to ...
- 3,277
4
votes
Watermark on Video implementation , Server or Client
There are two considerations here:
Adding a watermark takes quite a bit of processing power, especially for videos. It might be possible that a phone is capable of adding the watermark, but it will ...
- 72.5k
4
votes
Accepted
What's stopping Cloud Platforms from owning the whole deployment process?
Taking your example of AWS, it can indeed manage the entire deployment process, including deploying the full environment from Cloud Formation config files through to the source code of the application....
- 39k
4
votes
Fire aws lambda at a specific time
I had to tackle a similar problem and depending on the volume of scheduling that needs to happen, an elegant solution would be to use AWS Step Function with a Wait Step, see https://docs.aws.amazon....
- 41
4
votes
Accepted
Re-architecting CPU intensive Node application to handle multiple users
Nodejs is good for IO intensive tasks but may not be good for CPU intensive tasks. The reason is that it runs on the event loop, which runs on a single thread.
There may be several possible ...
- 3,142
4
votes
How to handle backpressure in message queue
How throttling helps to solve backpressure problem? If the TPS is consistent, Wouldn't it create the same problem even when producer retry after delay? At the end some producers will exhaust retries ...
- 12.4k
3
votes
Send multiple record types to a single Kinesis stream or not?
It depends on how your consumers process the streams.
Every consumer processes each type of event, independently
This might be a typical logging stream: you care that the events happen, but aren't ...
- 5,220
3
votes
Accepted
Messaging between microservices on AWS
"We are considering creating a queue for each microservice and publishing the messages to the queues via SNS."
Yes, then your queues can subscribe to the notification and you no longer need the ...
- 71k
3
votes
Amazon RDS read replica without support from application
When designing databases you have two basic choices: Scale-Up or Scale-Out. In the first case you use bigger machines (instances), for the second case you add more machines (instances) and this means ...
- 131
3
votes
Can a Database Scale Up?
Of course you can scale databases.
Also, with a strict interpretation of monolith and microservice there generally shouldnt be this kind of performance scaling advantage to microservices. A monolith ...
- 71k
3
votes
Accepted
What is the use of ENI in AWS?
ENI is a network interface, which embodies some configuration, which can include IP addresses, an elastic IP (EIP), various security groups (firewall rules), a MAC address, a source/destination check ...
- 164
3
votes
Accepted
Why should I use containers instead of deploying build artifacts directly to Azure App Services or AWS Elastic Beanstalk?
On a very high level, you are not missing out on anything crucial. Docker (and Kubernetes) provide a runtime environment, control plane, deployment, network abstraction etc. which are also provided by ...
- 2,867
3
votes
Why should I use containers instead of deploying build artifacts directly to Azure App Services or AWS Elastic Beanstalk?
A full answer to your question depends on the complexity of deployment. If you are looking to host a single ASP.Net (core or classic) website, with no additional services, then I would recommend the ...
- 45.8k
3
votes
AWS Architecture: How to search in Dynamodb?
Which AWS tool should I use for searching the DynamoDB?
You don't. DynamoDB is designed for quick and efficient lookup by key and by key only1. If you are attempting to search DynamoDB as any part of ...
- 22.9k
3
votes
Accepted
Foreign Keys and Eventual Consistency
So the foreign keys come from a source system (maybe an OLTP database) where they are immutable, and the metrics database (a kind of analytics DB) reuses them.
For such a situation, I would say it is ...
- 201k
3
votes
In Publish-subscribe pattern can a microservice publish message for itself
We did this in one of our services. It felt cleaner because everything was going through the same interface instead of having a separate 'internal-to-the-service' interface that behaved slightly ...
- 147k
3
votes
Accepted
Multiple Lambdas, one database?
Recommendations like “every service should have its own database” are just guidelines.
The point of this guideline is that if a database were to be shared between multiple services, this would obscure ...
- 133k
3
votes
Do Lambdas(AWS) stop making sense after a certain number of functions?
Working with Lambdas really requires some deployment automation tooling to manage all of these things for you. At my last job, we used Serverless Framework, but after a year I was starting to feel ...
- 131
2
votes
Accepted
Deliver message to websocket client connected through load balancer
Why not just broadcasting the message to all the nodes, and then let the nodes chose the messages they want to forward through WebSockets to clients?
If you have a lot of messages (that much that the ...
- 135k
2
votes
Best way to manage Git branches in different environments
The typical way to do this is to have a branch for each feature to merged in to master when done. For production releases, make a release branch off master. For e.g. nightly builds, just run the ...
- 36.8k
2
votes
Best way to manage Git branches in different environments
You can do it with individual developers working in branches and then merge into master as required, then just deploy from master into each of your environments.
Or you can do it via top level ...
- 201
2
votes
Cost of Cloud Vendor Lock-In
It depends on the importance of the software using the services. In this sense it is not any different from any other type of technology "lock-in" (not necessarily vendor specific). If the software is ...
- 3,932
2
votes
Autogeneration cloudformation. Good or bad
I've been using the cfndsl RubyGem for about 9 months now in a couple of projects, and have been very happy with it for the following reasons:
Comments
Invaluable as your scripts get more complex.
...
- 5,220
Only top scored, non community-wiki answers of a minimum length are eligible