Questions tagged [aws]

AWS refers to Amazon Web Services.

Filter by
Sorted by
Tagged with
9 votes
5 answers
507 views

Re-architecting CPU intensive Node application to handle multiple users

A few years ago, I wrote an application that allowed users to upload a file (it's a very specific file type) to a server. It then sends instructions to the server on how to visualise the data, and the ...
user avatar
  • 79
8 votes
3 answers
4k views

How to handle backpressure in message queue

I am designing a AWS web service which is going to get 1000 TPS from devices(Android) and it has dependency on multiple downstream services. The usecase is to hit this service periodically from device,...
user avatar
8 votes
2 answers
3k views

DB migration strategy for docker containers in AWS ECS

I am working on creating a set of microservices in Docker (using .net core and AWS RDS as the backbone, but that is not relevant). As part of the deployment, the old container and new container co-...
user avatar
  • 364
7 votes
1 answer
3k views

Invoking lambda directly from S3 vs going via SQS

I'm relatively new to AWS. I'm working with what I think is a common pattern: Put file in S3 bucket Do something with said file in Lambda function I see two options for making this link (ignoring ...
user avatar
7 votes
1 answer
7k views

Best way to notify the client in real time that their queue (e.g. SQS) job has finished?

Current Scenario Our application allows users to upload (Amazon S3) and manage their files through our interface. Currently those users can download the files directly from S3/Cloudfront through our ...
user avatar
6 votes
2 answers
5k views

Which is the best option to exchange large amount of data in a micro-service architecture?

Application I am working on requires text contents to be extracted from various proprietary document formats like Microsoft word documents (doc, ppt, xls ), pdf and etc. I am planning to implement a ...
user avatar
6 votes
3 answers
297 views

Server architecture for short bursts of ~150 parallel CPU-heavy subqueries

The client sends a query (a few hundred characters) to the web service. This query can be split into 20 to 150 subqueries with a simple regex. Those subqueries can then be computed independently and ...
user avatar
  • 163
6 votes
4 answers
7k views

Building a distributed system on Amazon Web Services

Would simply using AWS to build an application make this application a distributed system? For example if someone uses RDS for the database server, EC2 for the application itself and S3 for hosting ...
user avatar
  • 6,528
6 votes
0 answers
360 views

Rails on AWS Elastic Beanstalk - limitations

I have been looking at AWS Elastic Beanstalk service for the deploymeny of my rails application, and I came to think it was actually a bad idea given some specific requirements of my application. But ...
user avatar
5 votes
1 answer
5k views

Does AWS Kinesis guarantee a specific type of ordering?

From AWS documentation: A Kinesis data stream is an ordered sequence of data records. Each record in the stream has a sequence number that is assigned by Kinesis Data Streams I don't see any ...
user avatar
  • 277
5 votes
0 answers
2k views

Getting AWS SNS Messages into Azure Service Bus

I have a similar question to one already asked about SQS to Azure Service Bus - https://stackoverflow.com/questions/51650073/consume-aws-sqs-and-publish-to-azure-service-bus. I want to get messages ...
user avatar
4 votes
4 answers
297 views

Do Lambdas(AWS) stop making sense after a certain number of functions?

I am working on a project and created multiple lambda functions and now I am thinking about whether I made the correct decision to go this route. Reasons I chose lambda: The App will have spiky ...
user avatar
  • 149
4 votes
1 answer
579 views

How to organize projects in AWS?

In our team, we are using AWS as our main cloud provider and currently, we have 3 projects hosted on their platform. We are about to have 2 more projects in the next weeks, but first, we want to ...
user avatar
  • 201
4 votes
3 answers
644 views

how to pass credentials to api gateway securely?

I have a code at my local machine, with which I want to call an AWS Lambda function. I have configured AWS API Gateway as doorway to Lambda function, but I am concerned about security when sending ...
user avatar
4 votes
1 answer
643 views

Sharding and application routing cross region on AWS

We current have a very simple Multi Tenant monolith, with a SQL SERVER backend (Self hosted on EC2 on AWS), and multiple application services talking to one DB behind an Classic AWS ELB. Our database ...
user avatar
4 votes
1 answer
687 views

How should AWS Lambda functions subscribe to domain events?

I'm doing my first Domain Driven Design (DDD) implementation. My architecture is comprised of .Net Core for defining my domain model, services, and building AWS Lambda functions. I am using Lambda ...
user avatar
4 votes
1 answer
193 views

Robust way to aggregate results from 2 AWS lambdas for a SPA

I'm new to AWS serverless applications and am looking for something like ocelot request aggregation on the AWS serverless stack. Assume I have two AWS lambdas that return data needed by a SPA: A and ...
user avatar
  • 149
4 votes
2 answers
690 views

Where should authorization live in an application using API gateway/lambda and openid?

For an application which: Will use Google OpenID for account creation & authorization I plan on allowing any user to "create an account" which is why I want to use Google for this AWS API ...
user avatar
  • 12k
3 votes
2 answers
214 views

Foreign Keys and Eventual Consistency

We have a project that loads metric data from various etls and services into an AWS SQS queue, where that data is processed and inserted into a metrics database, also running on AWS RDS. When we ...
user avatar
  • 2,343
3 votes
1 answer
618 views

AWS Multitenancy Solution with RDS?

I am looking for the best way of implementing a Multitenant App in AWS. Imagine I am an eCommerce business that has several eCommerce clients and those clients have other clients that are the ones ...
user avatar
3 votes
1 answer
1k views

AWS SQS FIFO Limit Workaround

According to the AWS feature page for SQS: FIFO queues support up to 300 messages per second Standard queues support a nearly unlimited number of transactions per second (TPS) per API action. I'm ...
user avatar
3 votes
1 answer
772 views

DynamoDB - Event Store on AWS

I'm designing an Event Store on AWS and I chose DynamoDB because it seemed the best option. My design seems to be quite good, but I'm facing some issues that I can't solve. The Design Events Events ...
user avatar
3 votes
2 answers
282 views

Serverless architecture: How to deploy changes in common business logic?

I've been hearing lately more and more about serverless architectures which are based on products such as AWS Lambdas, Azure Functions, Google Cloud Functions, etc. I understand the advantages of ...
user avatar
3 votes
1 answer
112 views

AWS Serverless Microservice design

I am building an open source task management and note taking service that runs on AWS. I'm building it mostly as a hobby in my spare time and will eventually ship a mobile app that consume it into an ...
user avatar
3 votes
0 answers
378 views

Examples for "serverless" Java service for AWS Lambda

Doesn anybody have practical experience with implementing a (micro)service based on AWS Lambda? My team is currently working on an MVP. We started with a monolith and that has worked very well. It's ...
user avatar
  • 489
3 votes
0 answers
783 views

How do I implement Socket.io Leaderboard using AWS Dynamo DB cost-effectively?

Every time a user submits a score, Socket.IO causes all the users get their leaderboard updated. This causes every single user connected through Socket.IO to read from the Dynamo DB database (every ...
user avatar
  • 31
2 votes
2 answers
331 views

Allow users upload CSV files, and embed the data for a simple API

Looking for the correct architecture, that should take into consideration - future edge-cases, bugs and pitfalls, performance, cloud pricing, ease of building and maintaining and security. I have a ...
user avatar
2 votes
1 answer
99 views

Will hosting documents on AWS reduce our load?

Due to partnerships, we are required to host our website through our partner company. However, we had an idea that perhaps we could offload the document storage to an AWS document server. The ...
user avatar
  • 429
2 votes
3 answers
589 views

Securing API for third party use

We have a set of microservices and would like to expose endpoints from a subset of these for third parties to use. To this end, we will build an API Gateway that acts as the access control mechanism ...
user avatar
  • 165
2 votes
2 answers
2k views

Retry a failed REST API request - Java/Postgres

We have a REST API which calls a third-party REST API to Send Emails. The Third Party API is not super reliable and randomly fails every now and then with a 500. Our Clients do not want to retry at ...
user avatar
  • 139
2 votes
1 answer
1k views

Messaging between microservices on AWS

Here is the scenario: We are running a microservices style architecture on AWS using API Gateway and EC2 instances. Let’s say there are five services: Photo Service, Topics Service, User Posts ...
user avatar
2 votes
1 answer
2k views

What is the difference between AWS Kinesis and AWS Logstash?

I've been having the hardest time understanding the difference between AWS Kinesis and Logstash. From the description of Logstash it says Logstash is a light-weight, open-source, server-side data ...
user avatar
  • 123
2 votes
1 answer
424 views

Cost of Cloud Vendor Lock-In

Consider a greenfield development situation where cloud tools are being considered vs. in-house solutions, in the vein of AWS SQS vs. self-hosted Kafka, ECS vs. Mesos-Marathon, Lambda/Azure Functions ...
user avatar
  • 3,108
2 votes
1 answer
1k views

Deliver message to websocket client connected through load balancer

The problem I have a number of clients connected via websocket to nodes of my web application through a load balancer. What I need is to deliver notifications on a per-user basis. My idea My idea ...
user avatar
2 votes
1 answer
397 views

Updating a video's view count. Should this happen client-side or server-side?

I built an iOS app similar to Vine, where a user can scroll a feed of videos that auto play. All of these video have a View_Count field in the backend to keep track of how many times the video has ...
user avatar
2 votes
1 answer
151 views

Serverless game server architecture

First post on the software engineering stack exchange, I hope the question fits the purpose of this sub. I'm building a serverless game server with the following architecture and was wondering if ...
user avatar
  • 129
2 votes
1 answer
196 views

Does putting a serverless application in a VPC make it more secure?

I have an application which uses lambda and fargate (task) containers for compute, EventBridge for communication, S3 and DynamoDB for storage and exposes data out via an API Gateway. From my ...
user avatar
2 votes
1 answer
1k views

How to automate my AWS spark script

I am new to AWS and i have learnt and developed code in spark -scala . My application basically merge two files in spark and created final output. I read both files (MAIN files and INCR files )in ...
user avatar
  • 131
2 votes
2 answers
1k views

Best practices in saving user data when using identity provider like AWS Cognito

So in my mind, the usual users table that we are used to having, is going to split in two: whatever data is within the Identity Provider the domain specific data (user.facialHairStyle) is going to ...
user avatar
  • 121
2 votes
1 answer
413 views

HA for two availability zones in AWS

I am trying to setup consul for service discovery. We have hosted our infrastructure on AWS Mumbai region (We use only one region). I was going through the consul documentation where it is mentioned ...
user avatar
2 votes
0 answers
200 views

Determining when to use Serverless vs Containerized application (AWS Lambda vs ECS) - Is Java Spring dead?

I work for an organization that heavily leverages AWS. There is a strong push that every team move from containers deployed on ECS to leverage AWS Lambda and step functions for (almost) every project....
user avatar
2 votes
1 answer
263 views

Should a Web Application File URL Have Public or Private Access

I am debating whether to give my files a public url or a limited private one. I am hosting various files for a mobile/web application. These will include product images and videos. Currently only ...
user avatar
  • 161
2 votes
0 answers
26 views

In a multi-region network, how do you organize requests through VPNs for atomic database transaction requirements?

I am trying to setup a multi-region (global) AWS system, where each region (which has multiple availability zones) has its own VPC. In theory these VPCs all use different portions of the private IP ...
user avatar
  • 2,412
2 votes
0 answers
375 views

Running a high availability PostgreSQL cluster on native AWS services only

Backstory: I am unable to use RDS, as I need to install cartridges in my PostgreSQL instances. I have been trying to pin down an architecture for PostgreSQL running on EC2 instances for a few days. ...
user avatar
  • 29
2 votes
0 answers
26 views

Deployment pipeline as a component of an application

I am writing a serverless application using AWS SAM. I am wondering, would it be a good idea to include in my template a definition of a CodePipeline (and CodeBuild for that matter) for deploying that ...
user avatar
  • 121
2 votes
1 answer
71 views

Ideal architecture for ML training -> API service workflow, with multiple models/services? [closed]

I'm planning to construct a workflow/environment for training and serving NLP classifiers, that follows something like: The model training system takes in annotated documents from a subset of a ...
user avatar
2 votes
0 answers
42 views

AWS codepipeline deployment

I currently have an AWS Codepipeline that is setup to automatically pull an Express, NodeJS API and deploy it with Lambda and API Gateway. It pulls the code from my GitHub repo production branch and ...
user avatar
2 votes
2 answers
1k views

Preventing abuse of API which is called via client side Javascript

I'm setting up an AWS API Gateway + Lambda micro-service that will return non-sensitive but user-specific data.This means that the responses from the service can't be cached and served to multiple ...
user avatar
  • 87
2 votes
0 answers
481 views

Designing large nanoservice architecture with AWS Lambdas

I'm moving from a coupled architecture to a decoupled architecture using microservices with AWS Lambda. Here is my current architecture: Each API Gateway route is linked to a specific Lambda, each ...
user avatar
  • 169
1 vote
2 answers
508 views

Why should I use containers instead of deploying build artifacts directly to Azure App Services or AWS Elastic Beanstalk?

Recently, I have been told by others to look into containerization of my stateless web applications (in this case .NET Core 2.x and 3.1). All of my dependencies are retrieved from public and private ...
user avatar
  • 226