5

I've been building a monitoring system that checks whether a device is alarmed or not, and if so, sends notifications to relevant users based on how long a device is in a particular state. Here's an example of the criteria I'm building around:

1.) If a device has previously operated in a non-alarmed state for more than an hour previously

  • a.) If the device has been alarmed for more than 15 minutes, notify level 1 users

  • b) If the device has been alarmed for more than 24 hours, notify level 2 users

2.) If a device reached a level one alarm state

  • a.) If a device has currently been operating in a non-alarmed state for more than an hour, notify all relevant users

To build this system I've had to construct a lot of nested conditional logic and track state changes in a cache I set up. My server continuously gets new data on a group of devices and runs their states through alarm logic. The problem is that this system is very brittle and any changes I make to the logic for one state influence the others because they are intertwined. E.g.:

var deviceState = {...}; // cachedData

if (deviceState.hasOperatedWithoutAlarmForHour === true) {
    if (currentStatus === 'ALARMED') {
        if (deviceState.previousStatus !== 'ALARMED') {
            deviceState.hasSentLevelOneNotice = false;
            deviceState.hasSentLevelTwoNotice = false;
            deviceState.alarmStateStartTime = new Date();
        }
        else if (
            deviceState.hasSentLevelOneNotice === false
            && (currentTime - deviceState.alarmStateStartTime) > 15_MINUTES_CONSTANT)
        ) {
            sendLevelOneNotice();
            deviceState.hasSentLevelOneNotice = true;
        }
        ...
    }
}
else {
    if (deviceState.previousStatus === 'ALARMED') {
        deviceState.hasOperatedWithoutAlarmForHour = false;
        deviceState.noAlarmStartTime = new Date();
        deviceState.hasSentAllClearNotice = false;
    }
    else {
        if (
            deviceState.hasSentAllClearNotice === false
            && (currentTime - deviceState.noAlarmStartTime) > ONE_HOUR_CONSTANT
            && deviceState.hasSentLevelOneNotice === true
        ) {
            sendAllClearNotice();
            deviceState.hasSentAllClearNotice = true;
        }
        ...
    }
}

Is there a better way to construct a state-based monitoring system that tracks time duration in a given state?

2

I might design this system to take advantage of some sort of persistent state, seeing as how it operates over the course of many hours, if not days. This would allow for exit and re-entry of the application without effecting it's course of action.

To enable that functionality the following may suffice:

  • Device Status (as of time of check), along with UTC timestamp.
  • Action Taken (user notified, along with reason: error / recovery), along with UTC timestamp

I would then write multiple queries to determine what future action I should take for any given device, take that action, and log the action accordingly. Logging the action would change the result of the query and indicate that the action is no longer required.

Your queries and determined action would pair similar to the following:

  • To determine required notifications for level 1 users find all devices where the latest alarm time is greater than the latest level 1 notification time and greater than the latest non-alarmed time + 15 minutes.

  • To determine required notifications for level 2 users find all devices where the latest alarm time is greater than the latest level 2 notification time and greater than the latest non-alarmed time + 24 hours.

  • To determine required notifications for non-alarm status find all devices where the latest non-alarm time is greater than the latest non-alarm notification + 1 hour

Send said notifications, and log accordingly. Repeat as often as is useful per service level agreements.

  • Thanks, this is a good solution. I'm going to add an answer showing how I'll implement it. – Copernicus Feb 8 '17 at 15:48
1

This is how I plan to implement the solution from Christopher. In my particular case, I need to test the alarm status of each device separately from each other, so I'm creating a series of conditional statements that test each action case. However, I've separated out each case and decoupled them from each other.

In the cached state data, I'm only storing the previous most recent status, the most recent state change times, and the most recent action occurrence times. I've removed any Boolean state data I had previously which bloated the state object and increased the complexity.

I've separated each action so that they can be triggered separately and decoupled them as much as possible. Some conditions I'm testing are inherently linked together, but I've minimized it as much as I can.

Previously I used Boolean flags to track whether certain state changes had occurred. I've eliminated this entirely, and now all the logic for triggering an action lives completely inside each action's conditional logic.

// DEVICE STATE
let {
    LATEST_DEVICE_STATUS,

    LATEST_NO_ALARM_TIME,

    LATEST_ALARM_TIME,

    LATEST_LEVEL_ONE_ACTION_TIME,

    LATEST_LEVEL_TWO_ACTION_TIME,

    LATEST_ALL_CLEAR_ACTION_TIME,

} = deviceState;

const NOW = Date.now();

// MANAGE ALARM STATE

// Update the alarm state start-times whenever
// a state change occurs
if (
    CURRENT_DEVICE_STATUS === 'ALARM'
    && LATEST_DEVICE_STATUS !== 'ALARM'
) {
    LATEST_DEVICE_STATUS = 'ALARM';
    LATEST_ALARM_TIME = NOW;
}
else if (
    CURRENT_DEVICE_STATUS !== 'ALARM'
    && LATEST_DEVICE_STATUS === 'ALARM'
) {
    LATEST_DEVICE_STATUS = 'ALARM';
    LATEST_ALARM_TIME = NOW;
}

// ACTION LOGIC

// If the latest alarm has lasted for 15 minutes, send
// a level 1 notice if it hasn't been sent
if (
    LATEST_ALARM_TIME > ( LATEST_ALL_CLEAR_TIME + 15MIN )
    && LATEST_ALARM_TIME > LATEST_LEVEL_ONE_ACTION_TIME
) {
    TAKE_LEVEL_ONE_ACTION();
    LATEST_LEVEL_ONE_ACTION_TIME = NOW;
}

// If the latest alarm has lasted for 24 hours, send
// a level 2 notice if it hasn't been sent
if (
    LATEST_ALARM_TIME > ( LATEST_ALL_CLEAR_TIME + 24HR )
    && LATEST_ALARM_TIME > LATEST_LEVEL_TWO_ACTION_TIME
) {
    TAKE_LEVEL_TWO_ACTION();
    LATEST_LEVEL_TWO_ACTION_TIME = NOW;
}

// If there is no alarm, a Level 1 alarm was fired, and the
// device has had no alarm for more than an hour, send a
// notice if it has not been sent
if (
    LATEST_NO_ALARM_TIME > LATEST_ALARM_TIME
    && LATEST_TIER_ONE_ACTION_TIME > LATEST_ALARM_TIME
    && LATEST_NO_ALARM_TIME > LATEST_ALL_CLEAR_ACTION_TIME
    && NOW > (LATEST_NO_ALARM_TIME + 1HR)
) {
    TAKE_ALL_CLEAR_ACTION();
    LATEST_ALL_CLEAR_ACTION_TIME = NOW;
}
  • This implementation tangibly reduces the cognitive overhead of the algorithm with respect to the initial implementation, IMO. :) – cwharris Feb 8 '17 at 19:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.