-3

I am a junior developer and my typical senior developer just got fired and while they try to replace him I shall ask you.

I have an API that returns some internal data of ours in increments of 200 per call. There are about 40,000 pieces of data that I need to grab each week or 200 calls that need to be made to this API in a for loop. That data then needs to be inserted into an SQL database.

I have no problems writing code to call the API or insert it into the database, but I am wondering about the architecture/infrastructure to use for this. The API needs to be called once a week, so standing up a server just for that seems excessive. But is 200 API calls and some 40,000 inserts into an SQL database excessive for a Lambda function? It seems like it would be.

6
  • "But is 200 API calls and some 40,000 inserts into an SQL database excessive for a Lambda function?" — the answer is to build the application and measure the performance, otherwise you will spend a lot of time solving the wrong performance problems. May 14, 2021 at 23:23
  • 3
    "I am wondering about the architecture/infrastructure to use for this." — probably too broad for this site, but search for scheduled jobs or cron jobs. Certain database vendors also support something like that. May 14, 2021 at 23:25
  • @GregBurghardt I am mostly just trying to avoid doing something insane to start out as I have no idea what I am doing really. May 14, 2021 at 23:35
  • 1
    Actually, the most insane thing you could do is not try something out. With the senior dev gone, you'll get a lot of opportunities to try things out, because they probably do not have anyone else. May 14, 2021 at 23:37
  • @GregBurghardt: I don't get your initial comment - this question does not seem to be about optimization, but about picking an infrastructure for a repetitive task.
    – Doc Brown
    May 15, 2021 at 5:44

1 Answer 1

1

Before thinking about an architecture, estimate your budget!

Have a look at this XKCD:

Is it worth the time?

(Source: https://xkcd.com/1205/)

Now, maybe this was intended to be a joke, but let's take that seriously: you have a weekly task. Preparing a script which you just have to start, and putting a weekly reminder into your calendar will take you, let's say, 2 to 3 minutes per week (including a check that the script was run successfully). If you could set up a server which runs that task, you will still have to monitor in regular intervals that the server is running. The server could also send you a status email each week, checking that email will take you ~1 minute/week.

In the end, you will save ~2 minutes per week, which, according to the former table, is worth around 8 hours of working in a period of five years. Your task is going about ~4 years, this makes 6,4 hours in total. This is your "budget", for which you

  • have to get some CPU time on a server

  • put the script into a cron job, and get this correctly running (don't underestimate debugging time)

  • operate and maintain this system over a time of four years, which may include fixing some unforeseen issues

To me, this looks pretty scarce (and definitely not cost efficient when you have to rent or buy extra an server), so doing this manually is probably cheaper than making some automatism. You should also estimate what this will cost when using a "lambda function". You may save some operating effort, but also have to pay some rent, and I don't know Amazon's prices.

I would recommend, of course, to do the math by yourself, insert your own numbers and check where it is taking you.

Not the answer you're looking for? Browse other questions tagged or ask your own question.