0

I have one microservice (in GO) that needs to read a whole table every hour, filter out some data and send it to another application.

So as an example, imagine I have the following model:

type User struct {
  ID uuid.UUID
  Name string
  Surname string
  Birthdate time.Time
}

And I need to read all the users of the table users and send them to another service. As a prototype I created a cron job that every hour reads the database for one single user.

My question is, now I need to extend that to all the users, and I am wondering what's the best option, either to have one cron job that reads all the users, or to have one cron job per user.

I am thinking more in the future in the sense that I would like to be able to stop "collecting" data for some users but not for others, etc.

My feeling is that it's better to have one cron job to read all the users of the database, but I would like to have second opinions on it.

The number of users can be ~1000.

3
  • You've not really listed pros and cons here, or what your concrete end goal is, which makes it hard to answer your specific train of thought.
    – Flater
    Dec 13, 2022 at 9:18
  • Hi Flater, thanks for your message. I agree it's a bit "vague" description in the sense that maybe we miss how much time (approx) to get an user takes, or how much time/cpu consuming is the manipulation of the data, but I wanted to keep it "open" to not to bias possible answers. I would say the pros of having multiple jobs is that maybe, it's easier to stop them on demand, but the big cons is that I will need to manage 1000 different schedules and jobs, which seems like a difficult thing to do.
    – Manuelarte
    Dec 13, 2022 at 13:41
  • 1
    which seems like a difficult thing to do. It does depend on the needs tho. If that is what you need, then that's what you have to aim for. Given how is written the question, I assumed you can easily treat users in batches so there's no need for the overhead of 1000 independent jobs. The ultimate goal of any piece of software is releasing us from repetitive, tedious, complex or heavy-load jobs. I find taking care of 1000 or more jobs a bit tedious, making the solution... not that convenient after all.
    – Laiv
    Dec 13, 2022 at 15:58

1 Answer 1

3

The second option won't scale well... if that matters. Managing 1000 different schedules seems like a lot of work for me. Leave alone tracking and debugging if something goes wrong. Even if each user had different schedules, I would find it hard to sustain option #2.

I am thinking more in the future in the sense that I would like to be able to stop "collecting" data for some users but not for others, etc.

Configuration.

You can set a table or config file with all the users you want to keep in synch and which ones to exclude. Translated into SQL, it would result in a large WHERE statement depending on how you approach the data load (white or black list wise.).

Note that, this might fall short if each user has to be synced at different paces or times. Right now, looks like you can treat all the users alike (in batches), but if each user has a different sync schedule, then it's likely you need more jobs. For example, 1 cronn checking out the state of the data in your DB and planning (dynamically) the execution of more jobs. One for each user that needs to be synced. But this wouldn't be incompatible with the configuration mentioned above. Both solutions would be complementary.

1
  • excellent answer. Thanks
    – Manuelarte
    Dec 13, 2022 at 13:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.