0

I have a list of 75000 websites that need to be monitored for uptime. Monitoring the websites involves making a HTTP request for the website every minute, and the website is said to be "up" or "down" based on whether we could successfully make a HTTP request and received a HTTP response. This kind of "polling" is the only thing I can do, as I do not have control of the hosts running the websites.

Initially, I thought of having a couple of nodes, each of which would be tasked with monitoring a subset of the websites. Each node would run a program designed like so:

isServerUp(httpClient, url) {
    time = Time.now()

    try {
        httpClient.get(url)
        status = true
    }
    catch (e) {
        status = false
    }

    // do some other stuff, like saving the
    // status to a database.
}

websiteChecker(url) {
    httpClient = HttpClient()
    t = Thread(isServerUp, httpClient, url)

    while (true) {
        t.run()
        sleep(60)
    }
}

main() {
    for (website in websiteList) {
        t = Thread(websiteChecker, url)
        t.run()
    }
}

Basically, the program creates "websiteChecker" threads for each website to be checked. Each of these "websiteChecker" threads spawn a new "isServerUp" thread, that checks if the website is up.

However, such an architecture would barely work, as the huge number of "websiteChecker" threads would cause high memory consumption and extremely high resource contention.

How can I design an architecture that would work well in this scenario, and be scalable as well?

  • Wouldn't it be nagios.org? Or are you essentially trying to architect a replacement for it? – Berin Loritsch Feb 14 '18 at 14:45
  • @BerinLoritsch I'd like to build such a system myself, just for the learning experience. – user2064000 Feb 14 '18 at 14:48
  • 1
    I think downloading existing free software and ensuring that your hardware is capable would be a good first step, that will get you up (before you worry about other people being up): geekflare.com/best-open-source-monitoring-software -- Next understand Availability, checking and receiving an answer in less than one minute only ensures 99.95% Availability: en.m.wikipedia.org/wiki/… is that good enough. You might write the software and build 10 Servers but unless you have the pipes hosting will cost you big $. – Rob Feb 14 '18 at 17:31
1

When you are working with a large queue of websites like this, you'll want a combination of non-blocking I/O and a finite thread queue. You'll find that the exact number of threads that would be optimal is a function of the number of cores in your CPU. Testing will let you find the optimum ratio. Even then, you may need to distribute work across machines.

That said, what you need is:

  • A master work queue that is thread safe
  • Multiple threads to process tasks
  • A mechanism for asynchronous communication
  • A way to pause tasks until they get a response or timeout

Depending on your implementation the details are going to be a little different. For example, Java's NIO libraries have a different way of handling the interaction than C#'s async/await calls.

Architecturally, what is happening is something like this:

Address pulled off queue
  ^       |
  |  Task Starts
  |       |   \
  |       |    Request started
  |       v            |
  |_More capacity?   Passively wait for response or timeout event
       ^               |
       |           Determine success and report
       |        ___/
       Task Ends

You won't be able to spawn a thread per URL. Most operating systems have a limit of the number of threads that can be spawned per user, and then there's the practical limit where your processor spends more time context switching than it does doing work.

You can effectively make thousands of connections at the same time when you aren't blocking. Your tasks are in standby mode until the event that provides the data wakes them up.

Going through 75,000 websites every minute is going to be a challenge. You'll either need to do the request less often or distribute the work between multiple polling clients that coordinate with each other. In practice, polling once every 15 minutes is probably sufficient for 90% of people's needs.

| improve this answer | |
  • Only one note: multiple polling clients coordinating one with each other can itself be a challenge. I'd add a central distribution entity (a central brain logic is always simpler than a distributed one) which would take care of splitting the workload into chunks and passing them to the polling clients and could also take care of monitoring these clients, re-distributing the chunks in event of outages, starting/stopping them as needed, etc. – Dan Cornilescu Feb 14 '18 at 23:20
  • 1
    A proper message queue like RabbitMQ or Apache Kafka can work for this purpose – Berin Loritsch Feb 15 '18 at 1:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.