Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [aws]

AWS refers to Amazon Web Services.

0
votes
0answers
11 views

Understanding aws implementation for transferring data (CSV) from S3 to redshift datastore

I know there are a number of articles only and tutorials that show using RedShift COPY to get a CSV file/content into a redshift database. My question is on implementation. My thought is to place a ...
0
votes
0answers
31 views

Getting AWS SNS Messages into Azure Service Bus

I have a similar question to one already asked about SQL to Azure Service Bus - https://stackoverflow.com/questions/51650073/consume-aws-sqs-and-publish-to-azure-service-bus. I want to get messages ...
0
votes
0answers
26 views

Organizing and importing Go code for AWS Lambda function

I have a Go library and binary that implements an AWS Lambda function, organized as: $GOPATH/ src/ myorg/ lambdalib/ |-lib.go cmd/ lambdafunc/ `-...
-2
votes
1answer
69 views

How to start designing a complex information system?

Recently I've been assigned to a new project to start with designing and visualizing business processes. Unfortunately I just can't wrap my head around this one, I understand what the client wants but ...
0
votes
0answers
62 views

Is there a place for a REST controller in serverless?

Is it redundant to have a REST controller in a servlerss application that has http events? e.g. Imagine a serverless application like this: functions: hello: handler: com.sandbox.serverless....
0
votes
0answers
22 views

AWS S3 - make one broad call or multiple specific calls?

I've been working on a service that uses AWS S3 to store and access files. A user will belong to an organization and will have specific privileges (E.g an admin can access admin files and organization ...
1
vote
1answer
135 views

What's stopping Cloud Platforms from owning the whole deployment process?

Although things have come a long way in recent times, in my workplace we're still talking about DLLs, containers, configs, substitutions, pipelines - all complexities when it comes to shipping code. ...
0
votes
1answer
32 views

How to route/implement port forwarding with AWS/Apache web server (httpd) with a Node.JS app on port 3000?

I'm trying to understand this and having a hard time. I usually use Ubuntu server, with Apache2, and this time I am not. So I'm a bit out of my element. I'm also a software developer and I don't do ...
1
vote
1answer
151 views

Serverless event-sourced architecture using AWS offerings

This is my first post on the Software Engineering Stack Exchange so let me know if something is wrong with it. I'm looking into the serverless offerings of Amazon to try to figure out if that is the ...
0
votes
2answers
127 views

How to build reports for an ecommerce webapp without slowing it down

We have a medium sized ecommerce website which is entirely custom built in using Java and MySQL database on AWS infrastructure. Over a period of past several years, our orders volume has grown ...
0
votes
1answer
69 views

FTP upload on AWS Elastic Beanstalk

We have our REST API running on AWS Elastic Beanstalk (NodeJS/Linux), there are some legacy devices that their only way to communicate with the world is uploading a file using FTP. These devices ...
0
votes
1answer
61 views

Co-relating messages in Kafka

Context We are building an application that acts like a hub for messages coming in from multiple Producers and it assigns to Consumers based on certain criteria. Need is to have the design highly ...
0
votes
1answer
172 views

Preventing abuse of API which is called via client side Javascript

I'm setting up an AWS API Gateway + Lambda micro-service that will return non-sensitive but user-specific data.This means that the responses from the service can't be cached and served to multiple ...
0
votes
1answer
68 views

Watermark on Video implementation , Server or Client

Below is the context, Right now i'm developing an android application, which will allow the user to upload the video and image to the server. I want to allow the users(his friends) to download the ...
0
votes
0answers
50 views

aws lambda service for name generation

I want to implement a lambda service that would provide me random names (generated using patterns, some hardcoded values and so on) by GET request. For name generation I just use the set of prepared ...
0
votes
1answer
183 views

Send multiple record types to a single Kinesis stream or not?

We have a bunch of services which are using AWS Kinesis to communicate via events. We have different events such as MasterDataUpdatedEvent, PriceUpdatedEvent or AvailabilityChangedEvent. Currently, ...
0
votes
0answers
25 views

notify application about uploaded file on s3

I've got the application running on EC2. It is a simple REST app that consumes JSON (and XML in the future). On S3 in a bucket I put some XML and JSON files. I want to deliver them to the REST ...
1
vote
1answer
89 views

Building a real-time API on a Kafka/Kinesis-centric structure

I am a beginner to the event-driven data architecture using Kafka / Kinesis as the centrepiece (currently Kinesis) and I have some questions regarding how to build a WebSocket API on such a structure. ...
0
votes
0answers
354 views

Building docker images in Gitlab CI and pushing to AWS ECR.

I want to do something fairly simply in my Gitlab CI pipeline - I want to build the docker image and push it to an AWS ECR repo. I've looked at this Gitlab CI guide and one of the suggestions it ...
2
votes
1answer
494 views

Does AWS Kinesis guarantee a specific type of ordering?

From AWS documentation: A Kinesis data stream is an ordered sequence of data records. Each record in the stream has a sequence number that is assigned by Kinesis Data Streams I don't see any ...
0
votes
0answers
91 views

Is MapReduce a correct framework for Extract, Transform, Load of data?

EDIT I am working on a project to update a legacy ETL infrastructure that supports a number of clients, each with a slightly different setup. Constraints that cannot be changed: Source data can ...
1
vote
0answers
13 views

What's the preferred way to section a dynamodb table by client?

Working on an application that uses DynamoDB for data storage, which is new-ish for me. There are two tables, both use a simple guid for their key. I need to divide data per client. Someone can sign ...
2
votes
1answer
399 views

How to automate my AWS spark script

I am new to AWS and i have learnt and developed code in spark -scala . My application basically merge two files in spark and created final output. I read both files (MAIN files and INCR files )in ...
-2
votes
1answer
58 views

GNU license and hosting source code

Currently working with a client and their lawyers to launch a web application. The website will be hosted in AWS on Debian Linux instances. Debian contains (among others) the GNU General Public ...
3
votes
1answer
363 views

Which is the best option to exchange large amount of data in a micro-service architecture?

Application I am working on requires text contents to be extracted from various proprietary document formats like Microsoft word documents (doc, ppt, xls ), pdf and etc. I am planning to implement a ...
0
votes
0answers
178 views

Synchronous communication between different components on Heroku

I'm creating a chatbot in Java for Slack which looks as follows: To understand this diagram, think of the "Bot" process as a Slack user that, instead of being a human, is a computer program. There ...
0
votes
0answers
57 views

Share AWS global resources with AWS China

I have following services deployed in AWS Europe. User management Contract management (User subscription) The data is stored in Amazon RDS(MySQL) and services are exposed via REST Apis. Challenge ...
1
vote
0answers
39 views

iOS Push Notifications Certificate Sync Cert to Servers

I'm not sure if this is the correct form to be posting this on Stack Exchange. If not I apologize in advance and hope someone can point me in the right direction of where to post. I have a AWS Load ...
0
votes
2answers
424 views

How can I create a Python microservice on AWS that both accepts REST connections and processes SQS messages?

Background: I'm working on a new project at work that will run in AWS. We're trying to use a modern microservice architecture and take advantage of cloud technology, but we don't have much experience ...
1
vote
2answers
285 views

NoSQL or SQL for my situation? [closed]

I'm sure this has been asked multiple times but for me it's more specific as I know what I'm building. I wrongly posted this to stack-overflow (now removed) so I'm hoping this is the best place to ask....
1
vote
2answers
390 views

Best practices in saving user data when using identity provider like AWS Cognito

So in my mind, the usual users table that we are used to having, is going to split in two: whatever data is within the Identity Provider the domain specific data (user.facialHairStyle) is going to ...
0
votes
1answer
122 views

How to organise development process with AWS DataPipeline?

I'm introducing a Redshift for my team and it becomes a challenge to make a solid development process, so I need some info how to deal with it. First, of all some info about current layout. We are ...
0
votes
1answer
405 views

How can you reuse code for similar functions on AWS Lambda?

I have 3 Lambda functions that do very similar things: scrape data from different sections of a website. Each function returns a collection of objects that have similar properties. The method of ...
-1
votes
1answer
47 views

Amazon RDS read replica without support from application [closed]

I'm very new to this,so please correct if I'm wrong. I've understood that with Amazon RDS one can create read replicas and use those for all read-only operations. This helps scaling the application a ...
5
votes
2answers
2k views

DB migration strategy for docker containers in AWS ECS

I am working on creating a set of microservices in Docker (using .net core and AWS RDS as the backbone, but that is not relevant). As part of the deployment, the old container and new container co-...
1
vote
1answer
468 views

Messaging between microservices on AWS

Here is the scenario: We are running a microservices style architecture on AWS using API Gateway and EC2 instances. Let’s say there are five services: Photo Service, Topics Service, User Posts ...
2
votes
1answer
222 views

HA for two availability zones in AWS

I am trying to setup consul for service discovery. We have hosted our infrastructure on AWS Mumbai region (We use only one region). I was going through the consul documentation where it is mentioned ...
0
votes
0answers
108 views

Non HBase solution for huge data that has update and delete in sequential manner

I have to design an application where there are around 5K structured base text files (file.txt) with data and format as below: Primary key is OrgId + ItemId OgId|^|ItemId|^|segmentId|^|Sequence|^|...
0
votes
1answer
78 views

Autogeneration cloudformation. Good or bad

at my company we are using AWS Cloudformation to manage our infrastructure. The templates are written in JSON. The problem is that the templates are getting bigger by the day 5k Lines. Company policy ...
4
votes
2answers
469 views

Where should authorization live in an application using API gateway/lambda and openid?

For an application which: Will use Google OpenID for account creation & authorization I plan on allowing any user to "create an account" which is why I want to use Google for this AWS API ...
1
vote
1answer
515 views

Integrate an external rest api with my web application

I have a service running on AWS exposed via a rest API. The service processes files. Clients of the service post a file to the api and are returned a Job ID. Clients can then request the status of ...
2
votes
0answers
295 views

Examples for “serverless” Java service for AWS Lambda

Doesn anybody have practical experience with implementing a (micro)service based on AWS Lambda? My team is currently working on an MVP. We started with a monolith and that has worked very well. It's ...
2
votes
1answer
305 views

Cost of Cloud Vendor Lock-In

Consider a greenfield development situation where cloud tools are being considered vs. in-house solutions, in the vein of AWS SQS vs. self-hosted Kafka, ECS vs. Mesos-Marathon, Lambda/Azure Functions ...
2
votes
0answers
242 views

Designing large nanoservice architecture with AWS Lambdas

I'm moving from a coupled architecture to a decoupled architecture using microservices with AWS Lambda. Here is my current architecture: Each API Gateway route is linked to a specific Lambda, each ...
-1
votes
1answer
87 views

AWS CLI > why is the JSON output indented?

Just reading through the AWS CLI docs, and I've noticed that all JSON output is indented (i.e. "pretty" formatted)... Doesn't this add a disproportionate amount of bloat to the responses? If so, why ...
6
votes
3answers
270 views

Server architecture for short bursts of ~150 parallel CPU-heavy subqueries

The client sends a query (a few hundred characters) to the web service. This query can be split into 20 to 150 subqueries with a simple regex. Those subqueries can then be computed independently and ...
1
vote
2answers
552 views

How to design authorization between docker microservices?

I have 2 micro-services. One is a Django API, the other is a worker. Each one of these lives in side a docker container. The API is a public API for the web front end. It handles CRUD operations for ...
1
vote
1answer
577 views

Deliver message to websocket client connected through load balancer

The problem I have a number of clients connected via websocket to nodes of my web application through a load balancer. What I need is to deliver notifications on a per-user basis. My idea My idea ...
0
votes
1answer
56 views

When is cloud BlobStorage used?

Let's say we have an app deployed to Azure. App has external files not bundled with it, files which are required for app to operate. They are seperate but are essentially part of the software. ...
2
votes
0answers
499 views

How do I implement Socket.io Leaderboard using AWS Dynamo DB cost-effectively?

Every time a user submits a score, Socket.IO causes all the users get their leaderboard updated. This causes every single user connected through Socket.IO to read from the Dynamo DB database (every ...