Questions tagged [aws]
AWS refers to Amazon Web Services.
149
questions
2
votes
2
answers
1k
views
Preventing abuse of API which is called via client side Javascript
I'm setting up an AWS API Gateway + Lambda micro-service that will return non-sensitive but user-specific data.This means that the responses from the service can't be cached and served to multiple ...
- 87
0
votes
1
answer
355
views
Watermark on Video implementation , Server or Client
Below is the context,
Right now i'm developing an android application, which will allow the user to upload the video and image to the server. I want to allow the users(his friends) to download the ...
- 381
0
votes
0
answers
77
views
aws lambda service for name generation
I want to implement a lambda service that would provide me random names (generated using patterns, some hardcoded values and so on) by GET request.
For name generation I just use the set of prepared ...
- 203
0
votes
2
answers
2k
views
Send multiple record types to a single Kinesis stream or not?
We have a bunch of services which are using AWS Kinesis to communicate via events. We have different events such as MasterDataUpdatedEvent, PriceUpdatedEvent or AvailabilityChangedEvent.
Currently, ...
- 199
1
vote
1
answer
261
views
Building a real-time API on a Kafka/Kinesis-centric structure
I am a beginner to the event-driven data architecture using Kafka / Kinesis as the centrepiece (currently Kinesis) and I have some questions regarding how to build a WebSocket API on such a structure. ...
- 111
1
vote
1
answer
926
views
Building docker images in Gitlab CI and pushing to AWS ECR.
I want to do something fairly simply in my Gitlab CI pipeline - I want to build the docker image and push it to an AWS ECR repo.
I've looked at this Gitlab CI guide and one of the suggestions it ...
- 2,463
5
votes
1
answer
5k
views
Does AWS Kinesis guarantee a specific type of ordering?
From AWS documentation:
A Kinesis data stream is an ordered sequence of data records. Each record in the stream has a sequence number that is assigned by Kinesis Data Streams
I don't see any ...
- 277
0
votes
0
answers
669
views
Is MapReduce a correct framework for Extract, Transform, Load of data?
EDIT
I am working on a project to update a legacy ETL infrastructure that supports a number of clients, each with a slightly different setup.
Constraints that cannot be changed:
Source data can ...
- 1,386
1
vote
0
answers
21
views
What's the preferred way to section a dynamodb table by client?
Working on an application that uses DynamoDB for data storage, which is new-ish for me. There are two tables, both use a simple guid for their key. I need to divide data per client. Someone can sign ...
- 111
2
votes
1
answer
1k
views
How to automate my AWS spark script
I am new to AWS and i have learnt and developed code in spark -scala .
My application basically merge two files in spark and created final output.
I read both files (MAIN files and INCR files )in ...
- 131
-2
votes
1
answer
93
views
GNU license and hosting source code
Currently working with a client and their lawyers to launch a web application.
The website will be hosted in AWS on Debian Linux instances.
Debian contains (among others) the GNU General Public ...
6
votes
2
answers
5k
views
Which is the best option to exchange large amount of data in a micro-service architecture?
Application I am working on requires text contents to be extracted from various proprietary document formats like Microsoft word documents (doc, ppt, xls ), pdf and etc.
I am planning to implement a ...
- 69
1
vote
0
answers
102
views
iOS Push Notifications Certificate Sync Cert to Servers
I'm not sure if this is the correct form to be posting this on Stack Exchange. If not I apologize in advance and hope someone can point me in the right direction of where to post.
I have a AWS Load ...
- 179
0
votes
2
answers
1k
views
How can I create a Python microservice on AWS that both accepts REST connections and processes SQS messages?
Background: I'm working on a new project at work that will run in AWS. We're trying to use a modern microservice architecture and take advantage of cloud technology, but we don't have much experience ...
- 3
1
vote
2
answers
499
views
NoSQL or SQL for my situation? [closed]
I'm sure this has been asked multiple times but for me it's more specific as I know what I'm building. I wrongly posted this to stack-overflow (now removed) so I'm hoping this is the best place to ask....
- 37
2
votes
2
answers
1k
views
Best practices in saving user data when using identity provider like AWS Cognito
So in my mind, the usual users table that we are used to having, is going to split in two:
whatever data is within the Identity Provider
the domain specific data (user.facialHairStyle) is going to ...
- 121
1
vote
1
answer
731
views
How can you reuse code for similar functions on AWS Lambda?
I have 3 Lambda functions that do very similar things: scrape data from different sections of a website.
Each function returns a collection of objects that have similar properties. The method of ...
- 121
-1
votes
1
answer
82
views
Amazon RDS read replica without support from application [closed]
I'm very new to this,so please correct if I'm wrong.
I've understood that with Amazon RDS one can create read replicas and use those for all read-only operations. This helps scaling the application a ...
8
votes
2
answers
3k
views
DB migration strategy for docker containers in AWS ECS
I am working on creating a set of microservices in Docker (using .net core and AWS RDS as the backbone, but that is not relevant).
As part of the deployment, the old container and new container co-...
- 364
2
votes
1
answer
1k
views
Messaging between microservices on AWS
Here is the scenario:
We are running a microservices style architecture on AWS using API Gateway and EC2 instances. Let’s say there are five services: Photo Service, Topics Service, User Posts ...
- 123
2
votes
1
answer
413
views
HA for two availability zones in AWS
I am trying to setup consul for service discovery. We have hosted our infrastructure on AWS Mumbai region (We use only one region). I was going through the consul documentation where it is mentioned ...
- 151
-1
votes
2
answers
177
views
Autogeneration cloudformation. Good or bad
at my company we are using AWS Cloudformation to manage our infrastructure. The templates are written in JSON. The problem is that the templates are getting bigger by the day 5k Lines. Company policy ...
- 179
4
votes
2
answers
690
views
Where should authorization live in an application using API gateway/lambda and openid?
For an application which:
Will use Google OpenID for account creation & authorization
I plan on allowing any user to "create an account" which is why I want to use Google for this
AWS API ...
- 12k
1
vote
1
answer
906
views
Integrate an external rest api with my web application
I have a service running on AWS exposed via a rest API. The service processes files.
Clients of the service post a file to the api and are returned a Job ID.
Clients can then request the status of ...
- 11
3
votes
0
answers
378
views
Examples for "serverless" Java service for AWS Lambda
Doesn anybody have practical experience with implementing a (micro)service based on AWS Lambda?
My team is currently working on an MVP. We started with a monolith and that has worked very well. It's ...
- 489
2
votes
1
answer
424
views
Cost of Cloud Vendor Lock-In
Consider a greenfield development situation where cloud tools are being considered vs. in-house solutions, in the vein of AWS SQS vs. self-hosted Kafka, ECS vs. Mesos-Marathon, Lambda/Azure Functions ...
- 3,108
2
votes
0
answers
481
views
Designing large nanoservice architecture with AWS Lambdas
I'm moving from a coupled architecture to a decoupled architecture using microservices with AWS Lambda.
Here is my current architecture:
Each API Gateway route is linked to a specific Lambda, each ...
- 169
-1
votes
1
answer
180
views
AWS CLI > why is the JSON output indented?
Just reading through the AWS CLI docs, and I've noticed that all JSON output is indented (i.e. "pretty" formatted)...
Doesn't this add a disproportionate amount of bloat to the responses?
If so, why ...
- 165
6
votes
3
answers
297
views
Server architecture for short bursts of ~150 parallel CPU-heavy subqueries
The client sends a query (a few hundred characters) to the web service. This query can be split into 20 to 150 subqueries with a simple regex. Those subqueries can then be computed independently and ...
- 163
1
vote
2
answers
1k
views
How to design authorization between docker microservices?
I have 2 micro-services. One is a Django API, the other is a worker. Each one of these lives in side a docker container.
The API is a public API for the web front end. It handles CRUD operations for ...
- 111
2
votes
1
answer
1k
views
Deliver message to websocket client connected through load balancer
The problem
I have a number of clients connected via websocket to nodes of my web application through a load balancer. What I need is to deliver notifications on a per-user basis.
My idea
My idea ...
0
votes
1
answer
59
views
When is cloud BlobStorage used?
Let's say we have an app deployed to Azure. App has external files not bundled with it, files which are required for app to operate. They are seperate but are essentially part of the software. ...
- 1,129
3
votes
0
answers
783
views
How do I implement Socket.io Leaderboard using AWS Dynamo DB cost-effectively?
Every time a user submits a score, Socket.IO causes all the users get their leaderboard updated. This causes every single user connected through Socket.IO to read from the Dynamo DB database (every ...
- 31
0
votes
1
answer
503
views
Securing Amazon DynamoDB from Over-the-web attacks
I'm in the process of building out an architecture that will heavily leverage DynamoDb. My organization is considering the possibility of storing sensitive data (config info) in DynamoDb, as well as ...
- 259
3
votes
2
answers
282
views
Serverless architecture: How to deploy changes in common business logic?
I've been hearing lately more and more about serverless architectures which are based on products such as AWS Lambdas, Azure Functions, Google Cloud Functions, etc.
I understand the advantages of ...
- 127
6
votes
0
answers
360
views
Rails on AWS Elastic Beanstalk - limitations
I have been looking at AWS Elastic Beanstalk service for the deploymeny of my rails application, and I came to think it was actually a bad idea given some specific requirements of my application.
But ...
0
votes
1
answer
949
views
Most elegant way to execute a AWS lambda function with preloaded parameters on a specific time
I am looking for the following workflow:
Add an entry to a table of my Postgres database with a end_date and state (with the following values: active or done) columns.
Execute a lambda at the exact ...
- 169
1
vote
2
answers
6k
views
Best way to manage Git branches in different environments
I had a basic question regarding using a git repository in different environments. I'm more of a Sys admin than a web developer, so the teams I've worked on before just worked right in the master, ...
- 119
1
vote
0
answers
136
views
Distributed Design to Speed Up Video Processing
Currently I have one web server running Django, which takes a series of images from a user, uploads them to an Amazon S3 bucket, and finally calls an Amazon Lambda instance, which runs a python script ...
- 111
1
vote
1
answer
843
views
Amazon Echo Development on LAN
I have followed the tutorial from Amazon to get started with the Echo. I made a skill and setup an application server on their AWS Lambda for basic testing.
I have a few questions about the Echo, ...
- 21
2
votes
1
answer
397
views
Updating a video's view count. Should this happen client-side or server-side?
I built an iOS app similar to Vine, where a user can scroll a feed of videos that auto play. All of these video have a View_Count field in the backend to keep track of how many times the video has ...
- 153
0
votes
0
answers
51
views
What are the best practices of scaling small scale webservices in AWS?
I've deployed an entreprise app in AWS, in the first time i was hosting it in t2.micro, the app is intensivily used in 15 minutes a day, with some light queries in work hours during the day, after ...
- 101
-1
votes
1
answer
439
views
How to connect mean.js to AWS S3
I am trying see how I can upload images and file to aws S3 via the mean.js framework. I am not sure how the backend is linked to S3. There are no resources online about this specific framework and ...
- 101
-1
votes
1
answer
676
views
Is there any harm in using amazon AWS as a development environment? [closed]
What modifications do I need to make to my workflow, and which skills do I need to learn in order to develop with Ruby On Rails on a Linux cloud server on Amazon Web Services (AWS)?
In particular, I ...
- 17
2
votes
1
answer
99
views
Will hosting documents on AWS reduce our load?
Due to partnerships, we are required to host our website through our partner company. However, we had an idea that perhaps we could offload the document storage to an AWS document server. The ...
- 429
1
vote
1
answer
2k
views
Implementing distributed Semaphore for AWS workload
We have a messaging gateway in AWS (Amazon Cloud) that comprises of a number of nodes (Linux servers) responsible for receiving, routing, despatching and escalating messages (emails, SMS' and voice ...
- 111
0
votes
1
answer
711
views
Backing up messages in S3 within a Storm topology
In my project we are trying to build up a KIND-of-a-lambda storm based architecture. The component would be responsible for indexing the site usage events so we expect a quite massive random load. The ...
- 111
7
votes
1
answer
7k
views
Best way to notify the client in real time that their queue (e.g. SQS) job has finished?
Current Scenario
Our application allows users to upload (Amazon S3) and manage their files through our interface. Currently those users can download the files directly from S3/Cloudfront through our ...
- 173
6
votes
4
answers
7k
views
Building a distributed system on Amazon Web Services
Would simply using AWS to build an application make this application a distributed system?
For example if someone uses
RDS for the database server,
EC2 for the application itself and
S3 for hosting ...
- 6,528