We are currently in the process of doing an infrastructure overhaul.

A bit of background of what our business model currently is:

We are an aggregator of bills and payment services for businesses. Utilities, mobiles and much more. We are making it easier for companies to connect to these services by just connecting to our APIs. Each of these bills and payment services has their custom script with different APIs. We have a software acting as a middleman, it processes the incoming request, triggers the correct script, waits for the response, and then format and return the response.

Initially, our middleware and scripts are on two different servers. We use elastic beanstalk to host our scripts. However, we face a lot of issues. Since elastic beanstalk auto scale depending on the traffic coming in. Here is an example of what did go wrong.

  1. There are three instances running
  2. low traffic detected, shutting down one instance
  3. Suddenly, 100 request coming in, 20 requests are routed to the instance that is shutting down
  4. the middleware that is waiting for the 20 requests receives empty responses.

We then transition to a single large EC2 that host our middleware and our scripts together. The requests come in, and the middleware will point to the API (our PHP scripts) that is stored locally, runs it and return it to our users.

This worked out great. However, we encountered a hardware failure after two years of using this approach which brought down our whole server for 2 hours. Now we are trying to mitigate this and make sure the scripts have a 100% uptime.

It is also not scalable since edits to the code are made by RDP into the EC2 and changing the code locally.

Currently, we are trying to use EKS / Kubernetes instead.

Since Kubernetes also auto-scale nodes, I'm afraid the problem we faced when using elastic beanstalk might happen again.

TLDR: To summarize, will kubernetes auto-scale affect responses that are in the midst of processing? Some of our scripts might take more than 5 minutes since our scripts are synchronous.

Any recommendation on how to improve this architecture are also welcomed.

  • 1
    If you're trying to jump straight from “we're editing PHP files on production servers via RDP” to “Kubernetes auto-scaling” you will not have a good time. K8s doesn't fix your problem by default (pod might be shut down before it is removed from the service) but this can be solved. The real problem is your 5 minute scripts. If you want auto-scaling, it should be safe for your server to fail/stop at any time. Before going k8s, go 12 Factor.
    – amon
    Jun 3, 2021 at 12:26
  • what do you mean our server should be safe to fail/stop any time? You mean to have a backup server? Jun 7, 2021 at 15:43
  • No, though you do need hardware redundancy if your goals include high availability. What I meant is that instances of your services should start and stop within a couple of seconds, and that abrupt shutdown of a service should not lead to any integrity problems – it should be fine if scripts are killed before they can complete. It is possible to have a grace period of 10 mins so that your scripts can complete while scaling down, but the preferable solution would be for the scripts to make regular checkpoints so that their work can be resumed on a different node, with only a few seconds lost.
    – amon
    Jun 7, 2021 at 18:25
  • well the reason is our scripts perform transaction on 3rd party APIs, if we executed a transaction and then encounter a failure, we would have no idea to verify if the transaction succeeded. This in turn will cause us to lose money. is there a way to not process any transaction during abrupt failure (hardware failure, computer hangs) or have zero downtime? or should we use serverless such as google functions / aws lambda Jun 12, 2021 at 14:30


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.