Questions tagged [aws]
AWS refers to Amazon Web Services.
168
questions
-3
votes
0
answers
23
views
Best Practices for Backing Up a Local Mobile Database to AWS S3 [closed]
I am developing a mobile application that stores user records locally on the device. To safeguard against data loss (e.g., due to device loss or damage), I want to implement a feature that backs up ...
- 95
1
vote
2
answers
274
views
Is it worth it to use a NoSQL database only for high availability and fast reading, while already using Postgres?
We have several Postgres databases on one AWS RDS cluster (for budget reasons apparently), sharing the same resources, and my team wants to use a NoSQL database alongside it, and store data there from ...
- 113
0
votes
1
answer
95
views
Designing a Kafka consumer that gets a high throughput of "redundant" events
Context:
We have 2 services - a user-facing API server and a background processing Kafka worker.
Both of these are hosted on AWS EKS, and we use AWS MSK for Kafka.
The job of the API server is to ...
- 101
0
votes
0
answers
58
views
Most efficient way to maintain versioned business rules files for an Amazon ECS service?
I have an Amazon ECS service that relies on configuration files that contain business rules. The business rules are changed periodically by non-programmers. Total size of the rules files is ...
- 131
0
votes
1
answer
183
views
Is it a good idea to serve a Single Page Application as a static site?
I've built a small single-page web application in React and seen that it's possible to serve the app as a static site on something like S3.
Previously, I considered using Nginx, but as this is lower ...
- 159
0
votes
0
answers
43
views
Looking for patterns for instantaneous data consistency for DB transactions that span multiple schemas in a Aurora postgres instance
We are currently developing a product in the AWS environment that requires instantaneous data consistency for DB transactions that span multiple schemas in a Aurora postgres instance. Each schema will ...
1
vote
2
answers
229
views
Using Fargate (serverless compute) to replace leadership election
I currently have a solution that requires leadership election for HA. It processes files as they arrive in an SFTP server, and Subscribe to an Active MQ feed; and therefore we don't want multiple ...
1
vote
2
answers
163
views
Creating associated resources before creating the main one in a REST setting
I'm not sure how common of a scenario this is, but what are some approaches to creating associated entities before you have the actual main entity they're supposed to be linked to? (basically having ...
-2
votes
2
answers
178
views
Quiz Site Software Architecture Question
I am currently writing an application that is going to be an adaptive quiz-like site for studying. The idea is a user is studying some topic and they are given questions on the site and enter their ...
- 11
0
votes
1
answer
146
views
AWS and cron job: improve logging, monitoring and concurrence
In my company, we use a separate cron server to run mission-critical background jobs that run on a single ec2 instance: the whole platform is then vulnerable to anything going wrong on this instance.
...
-1
votes
2
answers
120
views
Architecture for EOD (end of day) stock exchange prices
I need to work out the architecture for a NASDAQ frontend charting application (a desktop app in .Net). Note that this is NOT for real-time quotes.
NASDAQ provides an api that gives historical pricing,...
- 117
0
votes
0
answers
65
views
Fine-grained access control to folders in S3 buckets with AWS Cognito
I have a private S3 Bucket that contains a lot of folders (currently 1000 folders). The amount of folders changes (new ones are added, old ones deleted). Every customer has to register and log in to ...
- 119
0
votes
0
answers
305
views
Transactionally update a set of files in s3
I have a bucket with several csv files at differing paths. The problem is that I want to update these csv files transactionally. The s3 bucket should represent the "latest" consistent set of ...
- 345
1
vote
1
answer
612
views
AWS multiple SQS queues on one bucket without fanout
I am coming into a project that uses an Amazon S3 Bucket for a process.
For my application, I need to know (within a reasonable time - order of a few minutes, at most) whether a new file has beeen ...
- 161
0
votes
1
answer
177
views
How do I avoid reading from S3 every time the user updates a filter on the webpage?
I am building a web application where a user logs in, and uploads a CSV file with employee data to S3.
This is an example of the format of the file.
id
name
country
age
3243
John Doe
USA
48
Using ...
2
votes
1
answer
1k
views
In an event-driven system, what are the best practices around the number of events and size/atomicity?
If you can have one event with multiple "things to process" or one event per thing to process, what is considered the best practice? Bearing in mind, the former option may involve persisting ...
1
vote
1
answer
112
views
Moving Transaction Logic to Middle Layer: Pros and Cons
We have an event booking system that allows patrons to add themselves to a waiting list if an event is fully booked. When somebody cancels, the system moves the first person from the waitlist into a ...
- 13
1
vote
2
answers
893
views
Multi-tenant multi-region SaaS with per-customer subdomain
I am trying to figure out how to design SaaS system that offers subdomain per customer (e.g. <customer>.example.com), where each customer is on specific region.
Due to compliance and regulation, ...
- 149
0
votes
0
answers
56
views
How to achieve high performance routing to Kinesis streams?
I have consumer pools that consume only events matching a specific attribute (e.g. origin=WINDOWS, the other origin=LINUX).
Since ordering is important (ordering is based on instanceId, so this is the ...
user374484
2
votes
1
answer
289
views
What is the best way to design a heavy lookup web app?
I am facing a bit of a conundrum with how I can structure my application, as I try to balance speed, cost, and efficiency.
I currently have a website, algfinder.com, which solves states of rubik's ...
- 131
1
vote
1
answer
583
views
Is DynamoDB optimistic locking with one version attribute for each field in an item a valid design pattern?
It's common to implement optimistic concurrency control in DynamoDB by giving each item in the database a top-level "version" attribute and only allowing an update of an item to succeed if ...
- 111
0
votes
0
answers
224
views
How would I model my NoSQL database for a conversational bot
I'm currently developing a chatbot that will be used for booking trips. The particular party I'm interested in for now is the rider. I'm using AWS's DynamoDB, because it seemed to be the most flexible ...
- 105
1
vote
1
answer
3k
views
How could creating symlinks/aliases to files on S3 work, while still taking advantage of CDN optimizations?
I am trying to imagine a website/system where only 1 copy of a file is ever stored, but multiple users get their own custom link to the file, and it uses the CDN features. For this question we can ...
- 2,615
6
votes
4
answers
1k
views
Do Lambdas(AWS) stop making sense after a certain number of functions?
I am working on a project and created multiple lambda functions and now I am thinking about whether I made the correct decision to go this route.
Reasons I chose lambda:
The App will have spiky ...
- 169
0
votes
1
answer
263
views
Best practices to deal with realtime data in REST API, Postgres and Redis
Overview
Today we have a monolithic service that deals with realtime time data. The monolithic are composed by follow stack: Django + AuroraDB Cluster (Postgres AWS) + Varnish cache and cloudfront at ...
1
vote
1
answer
568
views
Should I use different Buildspec files for branch builds and deployment builds?
Very recently a discussion came up regarding the usage of different buildspec.yaml files, one for branch builds and the other for deployment builds and I was wondering, since after some research I ...
- 113
1
vote
1
answer
726
views
Storing large data shared between microservices running in containers at AWS
I'm dealing with an application that does the following:
we have a microservice running on EKS called by jobs that pull XML data off an external system. The data volume can be fairly large (700MB-1GB ...
- 1,086
0
votes
1
answer
648
views
Removing successful cases from SQS batch. Is it wrong?
I have been working on a lambda that is triggered by a SQS with a batch (let's say of 5) and has automatic retry enabled.
The most common problem with this kind of setup that I have seen is that, if ...
- 119
1
vote
1
answer
1k
views
Multiple Lambdas, one database?
Having read through various microservices literature, I've seen a common recommendation that a microservice should generally be fairly cohesive, owning its own data repository, and serving as the ...
- 121
0
votes
0
answers
78
views
Do you put all CRUD Java Handler classes into a single AWS SAM template?
When building REST API with AWS Lambda + Java, do you define all lambda functions in a single AWS SAM template or create a separate project for each CRUD operation?
Given that on AWS, each lambda ...
- 117
0
votes
1
answer
484
views
AWS Java Lambda Class Naming Convention?
I wonder if there is a naming convention or a best practice to name Java Classes when building AWS Lambda functions with Java for REST API?
For example, if I have a Java project that contains Lambda ...
- 117
0
votes
1
answer
77
views
Table design question - DynamoDB the right choice?
Our team has started to use AWS and one of our projects will require storing approval statuses of various recommendations in a table.
There are various things that identify a single recommendation, ...
- 101
0
votes
1
answer
882
views
Is it a good practice to add a range key to an AWS dynamo table even if there is no need in current specification?
I am using AWS dynamodb for my project.
One can choose to use a simple key (partition key alone) or a composite key (partition key + range key) when creating a dynamo db table. But once it is created, ...
- 391
1
vote
0
answers
126
views
List Git branch currently on each EC2
We have a handful of AWS EC2s set up for testing ongoing feature developments. I'm wondering if anyone knows an easy, automated way to keep track of which git branch we're testing on each EC2. Maybe ...
0
votes
1
answer
615
views
Approaches for getting variables from AWS Parameter Store into app
I'm converting an app to run on AWS Fargate and was going to use the Parameter Store to hold configuration values. Most of the information I've seen about it revolves around injecting Parameter Store ...
- 1,086
0
votes
1
answer
975
views
Refresh Token Storage on cloud best practice?
Background
I am building a web app that allows the user to integrate with multiple services like Google, Twitter, Github etc. using OAuth2.0.
Currently, I retrieve the refresh token on sign-in to ...
- 101
9
votes
3
answers
7k
views
How to handle backpressure in message queue
I am designing a AWS web service which is going to get 1000 TPS from devices(Android) and it has dependency on multiple downstream services. The usecase is to hit this service periodically from device,...
- 201
0
votes
0
answers
90
views
How should I handle keys in production
I'm creating an authorisation service, which does signing using a key.
For local development I was randomly generating a key on application start-up.
Now I'm deploying to the cloud (currently going ...
- 685
1
vote
2
answers
373
views
In Publish-subscribe pattern can a microservice publish message for itself
I am designing a somewhat complex flow in publish subscribe pattern using AWS SNS, SQS , Microservice and Lambda functions. To break the whole flow into multiple logical & smaller sub-flows I am ...
- 31
-1
votes
1
answer
724
views
Design pattern to sync local databases with the cloud [closed]
I would like to find the best way to solve the following problem:
I need a database for a software. The software needs to use the data to populate the UI and allow the user to interact with them (edit,...
- 101
0
votes
0
answers
22
views
AWS Serverless Service to trigger an event when something doesn't happen
I'd like to use some sort of AWS serverless technology to trigger an event when a clickstream stops receiving clicks. Imagine a website visitor clicking around a site. Within our current solution ...
- 347
0
votes
0
answers
43
views
Kubernetes auto scaling
We are currently in the process of doing an infrastructure overhaul.
A bit of background of what our business model currently is:
We are an aggregator of bills and payment services for businesses. ...
1
vote
0
answers
58
views
Designing an architecture for running multiple scripts
Our company is an aggregator of multiple payment services fragmented on the web. So instead of businesses integrating multiple services one by one, they will only need to integrate to our API, and we ...
0
votes
2
answers
208
views
How to build an event sourced system with AWS (Specifically, how to deal with side effects)
So the current setup is a typical event sourced system built on AWS: Lambdas exposed with API Gateway to get the orders (commands), the domain (aggregates) apply some validations after loading the ...
- 101
1
vote
1
answer
396
views
AWS CodeCommit - duplicate test and production repos
I am using AWS CodeCommit with CodePipeline. I have
two totally separate AWS accounts, one for test and one for production,
identical Code Pipelines setup in each account, each with a corresponding ...
- 347
3
votes
2
answers
569
views
Foreign Keys and Eventual Consistency
We have a project that loads metric data from various etls and services into an AWS SQS queue, where that data is processed and inserted into a metrics database, also running on AWS RDS.
When we ...
- 2,323
1
vote
2
answers
397
views
When creating a RESTful API in AWS, why would you use EC2 over a Lambda integration?
In AWS you can easily create a RESTful API using API Gateway. With a serverless approach, you would have API Gateway integrate with Lambda to handle your endpoint logic. This seems like the most ...
- 1,181
0
votes
1
answer
445
views
Large file uploading in chunks
How to upload large(mb/gb) of video files from client to app server?
The solution as of now I know is pretty simple and widely used is to break file in chunks at at client side and send http post ...
- 81
0
votes
1
answer
330
views
AWS Architecture: How to search in Dynamodb?
Firstly, I want to state that this is my first time trying to build an app. I am doing this for fun, mostly as an exercise to learn system design/architecture. The goal is to build an app that would ...
- 117
1
vote
0
answers
419
views
How to (simply) architecture a way to ingest multiple types of large files, process them, and send data in chunks to web services?
Note: All of this would be in AWS
Hi everyone,
What would you guys suggest for building something that:
Takes in several different input file types (ex: csv, json, jsonl,
xml, .gz, ...)
That can be ...
user384015