I need to design a system where there are many micro-services which we can think of as crawlers that poll 3rd party resource and if they find some new information they pass it to some other service which processes the data and saves it in some DB.
I guess that I need to have some more component - A coordinator. To spawn the workers, to check if the services are alive and functioning etc.
So we have three main components:
- Worker
- Coordinator
- Processor
At this point of the project, I'm not sure how intensive the job of the workers is (i.e. the polling interval, the amount of new information). Yet, I'd like the system to be able to scale.
Now for the technical part:
- How should the workers notify the processor about new data? Should I use a simple REST server (i.e. Apache Tomcat) or some Message Broker to deliver the data to some backend service?
- Is there a proper framework or tool for the coordinator? I don't want to reinvent the wheel and I want to follow best practices.
- Obviously, it's wasteful to use a different server for every job. How to decide how many servers are needed? Can it be determined dynamically? Can all workers live on the same server?
- As I'm writing this, I am realising that conceptually, we need to differ between jobs and workers. So I think the coordinator needs to set a worker for every new job. Is that make sense?
As I said, I'm really not up to inventing the wheel so I'd love to use the right tools for the job (yet, I'm not looking for the most fancier solution - I want to keep it relatively simple)
Moreover, as I said, I don't have the numbers to determine how intensive the work is, but I need to bear in mind that the system shall be able to scale easily. I am using Java.