The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Hot answers tagged

6

There are some important aspects you should consider first. Streaming Let's imagine the 100 MB file is received by the service A which transfers it to service B, which, in turn, uses service C to do the actual parsing of the proprietary format. The wrong approach would be for the services A and B to start sending the file to the underlying service only ...


6

We don't know what you're going to persist in the db (vs. deal with in memory), what write volume, what is in transactions, what read volume, what joins etc.. Further, we don't know about evolution of your application: A simple new requirement may blow a NoSQL database out of the water, while a SQL could easily accommodate. To add to that, it sounds ...


6

Since you are CPU-limited, you need to get your hands on 150 CPU cores, one for each thread. This rules out a single server, since a server of such proportions would be prohibitively expensive – and you don't really need it. Your general architecture with a common frontend that distributes work to multiple workers and combines their results appears to be ...


5

The term that I would use is "arbitrary" rather than "random". But your observation is correct: the sequence numbers merely indicate the sequence in which records were added to the partition. They are intended to allow consumers to continue reading where they left off (using an AFTER_SEQUENCE_NUMBER shard iterator), not to reflect the producer's sense of ...


5

AWS Organizations is the standard approach to managing a hierarchy of accounts. However, it does not allow billing to be consolidated at arbitrary points in the organization; it's a tree, not a graph. Given that, here is what I would recommend: Adopt an "infrastructure as code" tool to manage your users and permissions. This is the most important thing ...


4

You can indeed. In fact, you can simply have your Web server send links that look like yourbucket.s3.amazonaws.com/yourdir/yourfile.txt You can even use cloudfront as a cdn, and optionally protect your files from direct access. You'll have to do some reading on s3 as a minimum, and cloudfront as well if you want. Also depending on your framework you will ...


4

There are two considerations here: Adding a watermark takes quite a bit of processing power, especially for videos. It might be possible that a phone is capable of adding the watermark, but it will definitely degrade the user experience. The server typically has more processing power, and there the operation can be performed in the background while users ...


4

Taking your example of AWS, it can indeed manage the entire deployment process, including deploying the full environment from Cloud Formation config files through to the source code of the application. Code is pushed to git in Code Commit and then code pipelines will build the environment, compile the code, run integration tests and allow everything to be ...


4

I think the most robust approach is S3 -> SNS -> SQS -> Lambda SNS gives you that pub sub endpoint so you can attach more things to the event if required. With SNS you can publish a message once, and deliver it one or more times. An SNS topic is an access point that recipients can dynamically subscribe to in order to receive event ...


3

When designing databases you have two basic choices: Scale-Up or Scale-Out. In the first case you use bigger machines (instances), for the second case you add more machines (instances) and this means read-replicas usually. Unless you have a small application with no heavy database requirements, I would modify my code to support read-replicas. This is no ...


3

A change should only change that which is necessary. Removing unused columns in a DB schema isn't a necessary change, it's a non-functional, maintainability kind of change. So there's no reason to lock in those changes to occur in the same iteration of any service (or, for that matter, be part of a service deployment at all). While you certainly could ...


3

"We are considering creating a queue for each microservice and publishing the messages to the queues via SNS." Yes, then your queues can subscribe to the notification and you no longer need the message service. Essentially its fan out routing. Your microservices become queue workers However, lots of queues and notifications can become complicated. SQS + ...


3

Of course you can scale databases. Also, with a strict interpretation of monolith and microservice there generally shouldnt be this kind of performance scaling advantage to microservices. A monolith can be just all the microservices in one big package. It sounds to me like this person is using the term 'monolith' to imply a solution which is very tied to ...


2

The scenario itself: "a user views a video should increase view count" is a client side activity resulting in backend data changes. It is initiated by the user using the client app, so it is okay to put the code that initiates and increases of view count in the client app. However, here are some other/different ideas to consider: Keep using the client ...


2

No - there is no harm in using an AWS Elastic Compute Cloud (EC2) instance for your development platform. But you will have to learn some extra skills that you wouldn't need when developing on a local machine how to use AWS Identity and Access Management (IAM), how to use AWS billing (if you exceed the one year free micro instance), and how to use SSH. ...


2

I would design my interface and task master under two assumptions: Processing power is distributed I don't always have enough processing power available If the jobs are inherently long, and the client/end-user knows this, my preference is to make the task master respectful of "competing" tasks. I'll favor progress notifications and mechanisms for client-...


2

There are (at least) two lines of thoughts regarding distributed systems, and depending on your environment your system may or may not qualify as such: Computer Science - in this case, a distributed system solves an algorithmic problem such that each node does part of the processing, in some instances even without a controller coordinating the task. Usually ...


2

It depends on the importance of the software using the services. In this sense it is not any different from any other type of technology "lock-in" (not necessarily vendor specific). If the software is extremely important to the owners, financially or otherwise, risk mitigation would include isolation layers, especially for the "higher level" services that ...


2

I've been using the cfndsl RubyGem for about 9 months now in a couple of projects, and have been very happy with it for the following reasons: Comments Invaluable as your scripts get more complex. Related items are colocated in the source For example, the Output for a resource can be physically located in the source file next to that resource, rather than ...


2

Probably not the best but still an option to consider. Install aws cli dynamically during the build, and possibly save as cached artifact/dependency. pip install awscli --upgrade --user


2

It depends on how your consumers process the streams. Every consumer processes each type of event, independently This might be a typical logging stream: you care that the events happen, but aren't performing any processing that needs to look at multiple events. In this case combining the streams makes sense: you have a single reader per consumer, and save ...


2

Since you noted that your queries run slow when using the MySql Workbench, that's the first place to start. In Sql Server, it's possible to view the query plan for a query and see exactly how the database will process it. Furthermore, you can check to see which index (if any) is used. In the case of your sample query, your WHERE clause contains a ...


2

I don't think you are going to find any authoritative answers to this, although there are a fair number of articles about. You should probably anticipate (in the first pass, anyway) that any mutable state in your solution could have more than one writer. So you should anticipate that all mutable writes use some sort of predicate/validator to support a ...


2

I can give you a little advice based on the general situation you are in. The project is actually already in development, although I'm not in favor of that decision Whether you are in favor or not, it's a fact. I'm going to wager you can't stop it and even if you could, it's not going to end well (for you.) So basically, development is underway and you ...


2

Main grey area, what can I do when the route goes to the wrong Region/Db? Re-rout them to the right DB. Seems simple. But... Tenant 1 went to USA for a holiday and tried to access www.domain.com, based on geo routing, he will be routed to the USA datacenter, which in the database in USA will not contain his data. ...is just the tip of the iceberg. ...


2

One thing I've learned early on doing these sorts of complicated procedures is to clearly name and record the steps taken thus far. If something goes wrong (and something will), you want to absolutely be able to trace back to the last effective operation performed. The more insight and details you can provide regarding this, the better. If there are ...


2

Use a normal queue. Having used these before I would simply accept the very low probablity of a duplicate message. But you can easily protect against them by the addition of a database to your processor. SNS -> Routing Service -> Message Sending Lambda cloud Give each message a GUID and have the routing sevice write this to a RMDB table within a ...


2

Make a .net web service that receives the data submitted by the form. Add that data to a database, using transactions to maintain consistency After adding the data to the Database call a second FileExport FileExport reads the database, generates the file and writes it to the s3 bucket. FileExport also buckets incoming requests. So if you get 50 updates ...


2

I believe there is a 6Mb limit on the request/response, which would make it unsuitable. https://docs.aws.amazon.com/lambda/latest/dg/limits.html Ideally you want a resumable stream of data for large stuff.


2

There are several options. 1) If you have AWS credentials on the machine with your client, you could use basic SDK (e.g. API C# SDK JavaScript SDK) 2) If you want to use the API Gateway, you can use Lambda Authorizer (Documentation). Credentials must be obtained from somewhere (e.g. AWS Cognito). 3) One option is to use API keys. Generate key for your ...


Only top voted, non community-wiki answers of a minimum length are eligible