2

I need to design a system where there are many micro-services which we can think of as crawlers that poll 3rd party resource and if they find some new information they pass it to some other service which processes the data and saves it in some DB.

I guess that I need to have some more component - A coordinator. To spawn the workers, to check if the services are alive and functioning etc.

So we have three main components:

  1. Worker
  2. Coordinator
  3. Processor

At this point of the project, I'm not sure how intensive the job of the workers is (i.e. the polling interval, the amount of new information). Yet, I'd like the system to be able to scale.

Now for the technical part:

  1. How should the workers notify the processor about new data? Should I use a simple REST server (i.e. Apache Tomcat) or some Message Broker to deliver the data to some backend service?
  2. Is there a proper framework or tool for the coordinator? I don't want to reinvent the wheel and I want to follow best practices.
  3. Obviously, it's wasteful to use a different server for every job. How to decide how many servers are needed? Can it be determined dynamically? Can all workers live on the same server?
  4. As I'm writing this, I am realising that conceptually, we need to differ between jobs and workers. So I think the coordinator needs to set a worker for every new job. Is that make sense?

As I said, I'm really not up to inventing the wheel so I'd love to use the right tools for the job (yet, I'm not looking for the most fancier solution - I want to keep it relatively simple)

Moreover, as I said, I don't have the numbers to determine how intensive the work is, but I need to bear in mind that the system shall be able to scale easily. I am using Java.

1
  • Tool recommendations are off topic here - I'd recommend editing your question to remove that part of it.
    – Jules
    Aug 21, 2018 at 18:34

1 Answer 1

2

How should the workers notify the processor about new data? Should I use a simple REST server (i.e. Apache Tomcat) or some Message Broker to deliver the data to some backend service?

You can use HTTP (REST) calls for this and keep it simple. The question is how you know what the host is that you want to call. You could consider using something like Kafka. This is designed for event distribution. I would avoid a 'message broker' solution unless you have a very good reason for it.

Is there a proper framework or tool for the coordinator? I don't want to reinvent the wheel and I want to follow best practices.

Container orchestration such as Kubernetes or Docker Swarm seems like a fit. These are not written in Java but you can run Java in the containers.

Obviously, it's wasteful to use a different server for every job. How to decide how many servers are needed? Can it be determined dynamically? Can all workers live on the same server?

Again, this has containerization written all over it.

As I'm writing this, I am realising that conceptually, we need to differ between jobs and workers. So I think the coordinator needs to set a worker for every new job. Is that make sense?

I think so. It seems like a natural solution.

2
  • Thank you. I don't quite understand the difference. Both Kafka and REST API must have some host the crawlers needs to know in order to communicate with it.
    – IsaacLevon
    Aug 21, 2018 at 19:47
  • @yaseco Yep. I can't remember the specifics but there's a solution that they've come up with that is pretty well proven out. Like I said, you can use basic HTTP for this and work out a way to find the host(s), you just might want to look at Kafka because it seems to fit with what you are doing. Even if you were to use some sort of messaging system, if you need confirmation that the message was received, your push to the messaging platform will be synchronous. It's what happens after that that takes some design.
    – JimmyJames
    Aug 21, 2018 at 20:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.