Architecture for Web application to monitor remote servers

Question

So I'm a relatively new programmer, attempting to create a web application (ASP.net) to display the system information (E.G. Status of windows services, disk & resource usage and errors in event logs) of a number of remote servers for monitoring purposes (initially - I'd also like to keep the design open to the addition, in the future, of some basic remote control over these servers too).

I've started by beginning to create a small prototype but I've become a little confused in regards to the design of the solution.

My current solution involves the creation of a windows service that runs on each of the client servers (those being monitored), which periodically collects the necessary data from the local machine and then sends it as JSON via a POST request to a Web API hosted in IIS on the Monitoring server, which receives the data and stores it in an SQL Database. I now intend to create a separate ASP.net MVC project to host as a separate application in IIS on the Monitoring server and use this as my front end - to read from the DB and display the stats of the machines.

Is there anything blindingly wrong with the design specified above?

I'm also unsure on whether this design limits how far I could take this app, for example how I could add features to allow users to perform actions on the client servers via the web interface (E.G. manually requesting for real time system information, remotely controlling windows services, clearing event logs etc). Would it be a bad design to also host a Web API within the client windows services so that the server could talk to/make requests to all the clients too?

Sorry in advance if this post is too lengthy/ambiguous - Any advice and/or design suggestions would be greatly appreciated!

Answer 1

There is nothing "wrong" with an approach to design/deploy an agent to PC's which scrape logs and feed into a central source. But when I was recently asked to develop a solution that entailed polling PCs for certain health/configuration bits, I chose to go with a server that polls via WMI to get the desired stats. If the bits you need to gather are available via WMI (and they probably are) I would recommend that versus scraping logs.

I am currently polling about 60,000 windows desktops and servers globally for a company. I have a service account provisioned with the necessary rights, and four processes running on a windows 2008 box (one for each region), and dumping the results into MSSQL and then using SSRS as the presentation layer.

Once I was able to gather the data and provide views of the problem across regions/divisions, and server and desktop counts that were vulnerable, the problem quickly turned to how to change configurations on a massive number of hosts to close off the vulnerability, and to do that I just needed to change my WMI call from a "get" to a "set".

By leveraging WMI and putting the "agent" on a server I avoided the time required to develop and test a client package, and having that pushed out, etc. And when the requirement changed, which is often the case, I didn't have to go through that development/deployment iteration again. And WMI is here to stay, so my code will work as the desktops transform to windows 10 and servers to 2012. If log locations change, etc. I am not impacted. And since I didn't deploy anything to hosts I can't be blamed for performance issues etc. Calling WMI to get information is minimally impacting.

Scraping logs should be a last resort. If the platform you are collecting data provides an API, leveraging that API will always make your job easier and your solution scale, survive version upgrades, etc.