0

I'm designing a infrastructure to monitor websites and applications. The idea is to build an application which checks the user website/application with a certain interval (30 sec, 1 min, 5 min, etc.). It's important that the website check is close to the given interval. The goal of the infrastructure is to have a scalable infrastructure with a high uptime without a single point of failure (if that's possible). A monitor which is 99% online of the time online isn't acceptable.

So after some brainstorming I came up with the following infrastructure.

Infrastructure

This diagram contains all the different parts/applications of the infrastructure. Each application runs on its own machine.

At this moment the manager delegator application is a high risk. If that application is down, it cannot move websites to another delegator if one of the worker delegators is down.

So my question is if this infrastructure is reliable and scalable enough for monitoring thousands of websites. Assuming there are enough workers to do all the monitoring. Recommendations are welcome!

1

Consider a design that uses the Actor Model.

With the Actor Model, you can have several fault-tolerant agents, each of which is capable of monitoring one or more websites. The Actors can execute on any machine, anywhere in the world; all you need is some hardware to run them on. The software architecture is dictated by the actors themselves. So rather than having machines that are dedicated to specific tasks, you dedicate software agents to specific tasks that can be run on any general-purpose machine.

This allows you to start with one machine and scale up as needed by simply adding more machines, or even running the whole thing in the "cloud." One actor can even manage the total number of actors running.

Further Reading
Actor Model on Wikipedia
Akka (an Actor Model Implementation in Java)
Akka for the .NET Framework

| improve this answer | |
1

A few things you may want to consider:

  • Geographic distribution of workers. This may help determine if the outage is the site or the internet.
  • Design the tasks so the workers can continue work even if its delegator is unable to communicate. This removes the risk as the delegator only needs to communicate task changes.
  • How/where to aggregate the responses.
  • What kind of historical data is desired. This is necessary for uptime/performance statistics.

I would expect a server to be available at least 99.9% of the time. With minimal scaling of workers, it should be possible to increase availability significantly.

If you are doing downtime notifications, then immediacy of messaging may be important. Normally, only one notification per outage is desired. Consider delegating to an odd number of workers and using quorum to determine if a site has gone down.

There are existing systems that already do this kind of thing. On the open source side look at Icinga/Nagios.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.