System design for multiple workers

Question

I need to design a system where there are many micro-services which we can think of as crawlers that poll 3rd party resource and if they find some new information they pass it to some other service which processes the data and saves it in some DB.

I guess that I need to have some more component - A coordinator. To spawn the workers, to check if the services are alive and functioning etc.

So we have three main components:

1. Worker
2. Coordinator
3. Processor

At this point of the project, I'm not sure how intensive the job of the workers is (i.e. the polling interval, the amount of new information). Yet, I'd like the system to be able to scale.

Now for the technical part:

1. How should the workers notify the processor about new data? Should I use a simple REST server (i.e. Apache Tomcat) or some Message Broker to deliver the data to some backend service?
2. Is there a proper framework or tool for the coordinator? I don't want to reinvent the wheel and I want to follow best practices.
3. Obviously, it's wasteful to use a different server for every job. How to decide how many servers are needed? Can it be determined dynamically? Can all workers live on the same server?
4. As I'm writing this, I am realising that conceptually, we need to differ between jobs and workers. So I think the coordinator needs to set a worker for every new job. Is that make sense?

As I said, I'm really not up to inventing the wheel so I'd love to use the right tools for the job (yet, I'm not looking for the most fancier solution - I want to keep it relatively simple)

Moreover, as I said, I don't have the numbers to determine how intensive the work is, but I need to bear in mind that the system shall be able to scale easily. I am using Java.

Answer 1

How should the workers notify the processor about new data? Should I use a simple REST server (i.e. Apache Tomcat) or some Message Broker to deliver the data to some backend service?

You can use HTTP (REST) calls for this and keep it simple. The question is how you know what the host is that you want to call. You could consider using something like Kafka. This is designed for event distribution. I would avoid a 'message broker' solution unless you have a very good reason for it.

Is there a proper framework or tool for the coordinator? I don't want to reinvent the wheel and I want to follow best practices.

Container orchestration such as Kubernetes or Docker Swarm seems like a fit. These are not written in Java but you can run Java in the containers.

Obviously, it's wasteful to use a different server for every job. How to decide how many servers are needed? Can it be determined dynamically? Can all workers live on the same server?

Again, this has containerization written all over it.

As I'm writing this, I am realising that conceptually, we need to differ between jobs and workers. So I think the coordinator needs to set a worker for every new job. Is that make sense?

I think so. It seems like a natural solution.