Questions tagged [scalability]

Scalability is the ability of a system, network, or process, to handle growing amount of work in a capable manner or its ability to be enlarged to accommodate that growth.

Filter by
Sorted by
Tagged with
0
votes
0answers
17 views

Scalable database design review

I'm doing some research on how to scale our current SQL database and need some advice with a possible solution. The goal being able to handle more data and doing it in a way that performs well. We ...
1
vote
2answers
157 views

How do atomic updates work at scale?

I am working on a weird ID system where I generate all the IDs in advance and need to atomically pop() one from the database, so no ID is read twice at the same time (and so used more than once). This ...
0
votes
0answers
18 views

Using WebSockets for event notifications along side a REST API with scaleable architecture

I have a REST API which can be called with HTTP requests and is connected to a database to store resource state. There is a particular Parent resource that has a collection of Child resources and if a ...
-3
votes
2answers
49 views

How should I handle a scalability problem during routing my requests from server?

I have faced a problem with the initial design of my system where sensitive information was being sent to the front-end and front-end was responsible for calling 3rd party APIs. As you have probably ...
0
votes
0answers
8 views

Scale spring-batch instances executed from within a web container

I am re-designing our current ETL for file ingestion where spring-batch containers will be deployed on Kubernetes. Each file type is processed by a different job, which will be triggered on demand via ...
0
votes
0answers
16 views

Trying to understand NIC receive / send operations per second estimations for non-abstract system design

I am working through resources related to non-abstract large scale system design. I'm working through the google system design exercise in this video. In the example solution presented, a calculation ...
-1
votes
3answers
75 views

Can I manage thousands of concurrent connections with a non-Node stack?

I'm developing a social network with Django (Python) + Postgres SQL. My site will have a chat feature so that users can communicate to each other in real-time, and the communication will be only from ...
-4
votes
1answer
47 views

application can not handle certain amount of RPS when 3k+ sessions connected

i have a problem in my scalable Pub/Sub application that it can not take more than 30 RPS when big amount of sessions are open against it. first i will explain the application structure. the ...
0
votes
1answer
55 views

How to make sure all of the nodes process one API request behind a load balancer

I have an architectural question. I have an application which is subscribed to a log compacted Kafka topic. I have to process each event and store into a persistent datastore. I am planning to run the ...
0
votes
0answers
35 views

Batch processing : Solutions for distribution of workloads

I work on product that is a multi-tenant cloud solution. When it comes to repeatable batch processing we have a set pattern. We configure a job to wake up and start executing its logic at regular ...
-1
votes
2answers
83 views

Developing your application as microservices or as a monolithic app - which is cheaper when deploying and scaling?

I am developing an application, and want to implement a certain functionality. I find that this can be done in the stack I am using, however it's hard to implement. I can create a microservice with ...
1
vote
2answers
113 views

Designing a scalable microservice: how to prevent one instance from performing an action if another instance allready did?

As I am moving a part of the monolith app logic to a microservice, I am standing before a problem with scalability. Currently, the main monolith runs on different instances, and has some scheduled ...
3
votes
2answers
224 views

Architecture for real time updates from the data in file system

I am trying to revamp my legacy application to make it scalable and performant. Its current architecture is something like this Consider a short-lived script that gets invocated 500k+ times a day, ...
3
votes
3answers
209 views

How can I scale a database that requires that writes are consistent?

Suppose I have a table that saves plane ticket purchases. I need to keep track that I don't oversell so when a user purchases a ticket it needs to be stored in that table and other customers need to ...
2
votes
1answer
47 views

How does a push-based(write fanout) feed system handle unsubscribe logic, message erasure and new following user's message

I am curious about these three questions when building a push-based(write fanout) feed system: How does it handle the unsubscribe logic? To me, it seems to me that we has two choice: delete or keep ...
-3
votes
2answers
77 views

How to scale an application where every read results in a write?

So I am trying to design a simple URL shortener application where every time a URL is queried, it is going to update the number of times it has been queried. I'm thinking of using MongoDB and I am ...
1
vote
1answer
61 views

Is my server design safe regarding multiple threads and concurrent database reads/writes?

I'm making a chat server using sockets and a MySQL database, and after it's working I want to expand it to become a more complex game server. I want to know whether my design is missing anything. ...
3
votes
4answers
475 views

Should we share a database between two 'Microservices'?

We're implementing a service that exposes data related to a particular part of a business. It will pull data in from different sources, do some ETL, and store it in Redis. It will expose this data ...
0
votes
1answer
67 views

Is event-sourcing and event-driven design works in scalable architecture for a wallet?

considering you need to design a simple wallet service that keep tracks of transactions, and give balance for each account. I started designing the solution using event-driven architecture, so every ...
5
votes
4answers
852 views

Should data transformation be on the front or on the back end in this scenario?

Let us say I have some data in a Json format. Let us say the Json format is the following: {title, resources[] } The resources array contains information about a graph. But it should be iterated ...
1
vote
0answers
25 views

How would i design processing twitter data

I am working on a project where a user can input different criteria which will be used to fetch tweets, lets call this action as TweetAnalysis. These tweets will then be sent to another internal ...
0
votes
0answers
69 views

What could be a good scalable alternative to polling data and then looping over a list of objects for triggering some action on those?

So I have some naive implementation for the following problem: We have a list of objects with action methods that have to be trigger at some value of interest. This value is polled (or is streamed) ...
1
vote
2answers
428 views

What is the best way to run multiple instances of a web app that requires state to be transferred between instances?

I'm working on a web app where users enter a "room." The state of these rooms is loaded in the server's memory at all times, and is synchronized with the clients via websockets. What's the best way to ...
0
votes
0answers
89 views

How can I economically scale to 100+ instances of my .Net Core app without using 1 container per instance?

Currently I have a .NET core application, which fires up a new thread to hold each customer account. Each thread instance: Is long running for many months or more Holds open SignalR Websockets and ...
-1
votes
1answer
50 views

Calling microservices after scaling

I am developing a microservice platform with Spring. So far I have a few microservices and a service discovery server where all microservices register themselves. I also have an Angular client ...
1
vote
2answers
447 views

Advantages of using a message broker for scaling websocket client-to-client communications

I'm designing a system where pairs of clients need to exchange messages proxied by a backend service. My initial plan was to use websockets and have the clients connect to a single websocket server, ...
0
votes
3answers
120 views

Using Kubernetes with an app that wasn't designed to scale

We have a .Net 3 tiered app that wasn't designed to scale horizontally. I've heard we can use docker to containerize our app and then put it on kubernetes. Is that enough for kubernetes to make it ...
2
votes
1answer
1k views

One process using std::thread, 2 physical CPUs, 4 cores each, parallelism level?

Suppose I have a one C++ process, and I want this process to run eight threads in parallel. And suppose that: I have a computer with two (2) physical CPUs. Each CPU has four (4) cores, so that's 4x2 ...
2
votes
1answer
247 views

How to better scale reads in a website: to CQRS or not to CQRS?

The company i'm working for it's rewriting a legacy web application and we are evaluating the better architecture to get the job done. The business domain is quite simple: there are few entities, ...
1
vote
1answer
65 views

Publish / Subscribe Paradigm Implementation Problems

Reading the final chapter 6 of the paper "The Many Faces of Publish/Subscribe" I stumbled across: Decoupling is a desirable property because it enforces scalability at the abstraction level,...
6
votes
2answers
528 views

Scalability: Millions of tables vs one big master table and millions of views

We have to migrate an old (million LOC) system from SAP ADS (formerly Sybase) ISAM based (so called free tables) based system to a PostgreSQL database. The system implements kind of multitenancy ...
0
votes
1answer
77 views

My local server is getting close to maximum resources, I want to add another server.

So, first let me explain my current setup. I have an unmanaged ethernet switch which is connected to my intel nuc (server A) and an external had drive. Server A is running: Ubuntu Docker Gitlab ...
1
vote
1answer
695 views

Design scalable solution with queue

Assuming the following scenario: A web application that receives inputs from the user and process the inputs through an algorithm which takes between 5 to 25 minutes (depending on the inputs) and ...
3
votes
1answer
252 views

Sharding and application routing cross region on AWS

We current have a very simple Multi Tenant monolith, with a SQL SERVER backend (Self hosted on EC2 on AWS), and multiple application services talking to one DB behind an Classic AWS ELB. Our database ...
2
votes
4answers
427 views

User calling REST APIs on behalf of a different user

Here's an example: GET /posts POST /posts GET /posts/{post_id} PUT /posts/{post_id} This api is being used by SUPER_ADMIN as well as CONTENT_WRITERS. Its working great. If user is logged in as a ...
1
vote
0answers
95 views

User level customised activity feed design

I am working creating api for activity feed of user. Feed contains different type of entities. Before that let me give you bit overview about the app. Its a video app where Users can ask questions ...
13
votes
7answers
1k views

Saving high-frequency events to a connection-limit constrained database

We've got a situation where I have to deal with a massive influx of events coming in to our server, at about 1000 events per second, on average (peak could be ~2000). The problem Our system is ...
1
vote
1answer
126 views

How should we approach layered-architectures that need large amounts of continuous, real-time, communication?

Background In much of the literature I've read online with respect to multi-layered architectures, many people describe how to create the simple application where: UI presents static model objects to ...
-2
votes
1answer
610 views

Shopping Cart Concurrency Question [closed]

I have a problem with my e-commerce site. There can be multiple simultaneous requests to add items in the shopping cart, I was moving towards the design where when an item is added to the shopping ...
0
votes
1answer
112 views

Guidance for Client side

We have created software for Rugged and mobile devices, and it gets deployed on the device. Admin has a provision to send an update to those devices. We use GCM or WNS for sending a notification to ...
0
votes
1answer
262 views

Co-relating messages in Kafka

Context We are building an application that acts like a hub for messages coming in from multiple Producers and it assigns to Consumers based on certain criteria. Need is to have the design highly ...
3
votes
1answer
131 views

Architecture for Large-Scale Matrix Multiplication: Distributed Architecture or One Strong Server

I've asked a question about Scaling Matrix Multiplication by CPU Cores on StackOverflow and it seems that merely adding more CPU cores to this problem won't reduce the time to do Matrix ...
2
votes
1answer
275 views

High scale architecture using Event Sourcing and Eventual Consistency

I want to share with you and know what your opinion/improvements are regarding a solution I was tasked with a technical interview I faced. The exercise: You need to create an API which will indicate ...
2
votes
1answer
62 views

Architecture and Database re-design for migration and support of multiple clients

I'm evaluating the migration of the following application's architecture: - Nginx + PHP + MySQL - Currently the infrastructure is scalable and redundant in the AWS cloud and It was designed to support ...
1
vote
1answer
298 views

Building a scalable Email Delivery system

I want to fix my current Email Delivery system which sends email using a third-party email provider and creates a record for each email sent in RDS. Functionally this is how the system behaves - A ...
0
votes
1answer
663 views

What would be the architecture for a polling-based monitoring service?

I have a list of 75000 websites that need to be monitored for uptime. Monitoring the websites involves making a HTTP request for the website every minute, and the website is said to be "up" or "down" ...
1
vote
1answer
1k views

Scalable design for a web application [closed]

My apologies if this is the wrong stack exchange site for this question. Please, let me know if I should ask this in a different place. I have recently went to a interview. The interview was for the ...
0
votes
1answer
110 views

scalable communication system

I am designing communication system that will be sending targeted based SMS (ads/offers) to its customer. The basic system with low load scenario works well, and the SMS sent are based on an event ...
0
votes
2answers
2k views

Best way to design file storage for image processing system

Would appreciate any recommendations regarding design a high-load image processing service. There are lots (tens of hundreds of relatively large images (~10...20 Mb each)) that to be processed using ...
4
votes
2answers
917 views

What time series database can support high cardinality?

I have a situation where I want to track page views, click events, etc. in a time series database but am having scalability issues in the case where I want to retrieve aggregated groups of data that ...