1

I'm implementing an application that's made of bunch of microservices. Each of them is fairly simple, focusing on its own task. Let's call two of them Service A and Service B. They are both implemented in NodeJs and expose a Rest API. Now I need to run couple of cron jobs (some at 1m interval, others at 5m, 1h and so on) and I need to pass data from one service to another. For example I need data from Service A, call an external API, process the data and POST it to Service B.

enter image description here

I thought of using another microservice called Task Master in this example, that would do this cron job and all the others (for all the services in the application), therefore being master of all the cron jobs.

  1. Is this a bad idea ? One microservice just for cron jobs ? And all the cron jobs ?
  2. What would be the best way to implement this ? Could it be another NodeJs app ?
  • One option is to have a message queue, and have the cron service send the correct message to invoke Service A or B appropriately. In this scenario, it can be any language you believe will be appropriate. Python, Java, (dare I say) C#, NodeJS, etc. – Berin Loritsch Oct 1 '19 at 20:08
1

This is a fairly common pattern. Common enough that there is existing software to accomplish a lot of what you've described. Look at kubernetes cronjobs, or celery periodic tasks, for example. Most serverless frameworks also support periodic scheduling. I would do a thorough review of existing implementations before trying to make your own.

  • Thank you, I've been playing around with Celery for some time now, looks promising. – Matis Oct 16 '19 at 12:42
1

Take a look Apache Airflow, also most of the serverless frameworks have something to accomplish that. For example a user a framework for Python and AWS Lambda named Zappa. This framework has a setting for scheduling jobs.

Another alternative to schedule jobs that are note very complex is to use Jenkins with some plugins you can build some simple pipelines.

1

In a Micro service environment, there would be a strong need to synchronize data across services. There are various options for it. Having a Cron job is definitely one of the approaches. Other approaches could be ETL, service periodically fetching the data it needs from other services and Messaging or Pub/Sub mechanisms. There could be other methods or synchronization but these seems to be the most popular and effective methods. When there are multiple ways of doing a job, it would be good to compare the pros and cons of various approaches.

ETL There are tools that could be configured to read the data base of service A directly and transform the data and insert/update in the database of service B. These class of tools are called ETL (Extract Transform And Load)

  1. (Advantage) There are many off the shelf products available that require only minimum work to get them integrated
  2. (Disadvantage) Now another application(ETL) knows the schema of the database of service A and service B. This reduces the independent deploy-ability of serivce A and serivce B. Any change in schema in db of serivce A will need re-deployment of service A and ETL. ETL jobs cause resistance in changing the schema of the databases.
  3. (Disadvantage) The tools available in the market are generally a bit expensive

Cron Jobs The cron job you have designed is similar to ETL in some aspects. But the significant difference is the job talk with application using language that is dictated by application and not by the database schema.

  1. (Advantage) The cron jobs are independently deploy-able. However we should take care so that the application contract is not broken.
  2. (Advantage) Very simple to implement.
  3. (Disadvantage) Generally the services are created based on the domain. Each sub domain is organized as a service to facilitate that functioning of the domain. Now cron job will have information about the data needs of various services. Over the period of time there is high likeliness that the number of cron jobs increase or the code size of the cron job becomes bigger and bigger. Either way it will become difficult to maintain the cron job.
  4. (Disadvantage) When cron job execution is in progress, the service could become less responsive due to the load.

Service B fetching data periodically from service A

  1. (Advantage) The services are independently deploy-able. However we should take care so that the application contract is not broken.
  2. (Advantage) Very simple to implement.
  3. (Advantage) All the requirments for the serivce B is coded in one repository. What data is needed is also coded in the repo of Service But
  4. (Disadvantage) There is a coupling with the data source (here Service A)
  5. (Disadvantage) Triggering the service B to fetch it's data could be a bit tricky
  6. (Disadvantage) When the service is fetching data, it could become less responsive due to the load.

Messagin/Pub-Sub Mechanism The service A publishes event/message to a message broker/eventing system such as kafka, RabbitMQ or Azure Service bus. The service B listens for these events/message. The event/message could contain all the data that is needed for service B, or the service B could query Service A to get more data on reception of the event/message

  1. (Advantage) Service A and Service B are highly decoupled
  2. (Advantage) The eventing/messaging infrastructure is dumb with no (not much of) business logic.
  3. (Advantage) Proper implementation of the system will ensure the responsiveness is good at all times.
  4. (Disadvantage) The systems are complex to implement.
  5. (Disadvantage) It will be difficult to implement service B using server less technologies such as Azure Function, AWS Lambda or KNative. The difficultly depends on the availability of the binding for the server less technology and eventing/messaging infrastructure.

I would personally prefer to start of Service B fetching data periodically from Service A, then move towards a eventing mechanism as the complexity increases in the system

1

What you're trying to achieve here is context mapping. Each microservice represents a context and you are moving(mapping) data from one context to another context. Ideally you want this communication to be asynchronous, so that the data between the contexts is eventually consistent. If you need it to be transactionally consistent then you should probably reexamine the boundaries of your contexts/microservices.

There are several ways of achieving context mapping. You can use a message bus, where ServiceA publishes a message that contains the data that ServiceB needs, and ServiceB subscribes to this message to recieve the data. However, this involves the use of infrastructure that may not already be in place and may take significant effort to implement.

Another way of achieving this context mapping is via polling. ServiceB can poll ServiceA for whatever data it needs from ServiceA. This polling can be a seperate process (it could be an AWS Lambda, Azure Function, or cron job), but it would still be part of the microservice ServiceB. Afterall, it's ServiceB that needs ServiceA's data. ServiceA, as you've described it, doesn't need anything from ServiceB. I guess the illumination here is that a microservice can be more than one process. I imagine, with what you've already described, this is the context mapping strategy that would best fit your scenario. No TaskMaster Service necessary, only a constituent process of ServiceB that handles retrieving the data it needs.

There are other context mapping strategies as well. See Vaughn Vernon's IDDD, for a full description.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.