Architecture for collecting metrics from process

Question

I have a class that oversees some process. At it's core, there is a process method which collects data and processes it. There are a bunch of methods for retrieving the results of the previous processing step.

To have a better overview, here is how it looks like currently:

class Main {
    private:
        // resource1, resource2 ...
        // result1, result2
    public:
        void process() {
             resource1.getData();
             resource2.getData();

             // process data
             // store results
        }

        T getResult1() {
            return result1;
        }
};

The users of this class are supposed to use it like this:

Main main(resource1, resource2);

while (condition) {
    main.process();

    output main.getResult1();
}

(I'd appreciate any comments about the current architecture too)

Now, the data comes in chunks, and the processing is done on a per-chunk basis. The idea is to insert some metrics in the class, that track the processing over time.

Requirements from the metric system:

• Should be able to store information over time
• Should provide some way of configuring the metrics used, as well as some functionality to clear a metric
• As separated as possible from the Main class
• The metrics need access to the results of the processing and various members of the main class
• There is a relatively small number of metrics (less than 10) currently considered
• The user of the main class should have access to the metrics all the time

One way would be to store the metrics inside Main (maybe inside a MetricManager class), and run them after each processing stage. The problem is that they need access to a lot of components from Main (maybe passing this as parameter would be an option?).

Answer 1

I want to build on part @JimmyJames's answer: having general containers for the specific metrics helps decouple this.

First, naming is important. Users of your Main class is going to be confused merely by the name. We associate main with the main entry point of a program. Is Main that? Seems more to be a Supervisor class. I roughly know what a supervisor class will do. Metric is also a bit fuzzy as I associate that with some metric. Maybe use Collector?

In order to separate the Collector from the Supervisor, you'll need to make explicit the intention of the user, i.e. to collect metrics. This can be done by forcing the user to make explicit this intention by e.g. providing Collector.collect with a signature that takes resource identifier of the thing being collected (a string), and the resource it is to collect.

Next, the results needs to be stored somewhere, and as @JimmyJames states, you probably need some aggregation, and you've also stated that you need to clear metrics from time to time. The Metrics class can be responsible for this. It can store metrics on specific resource identifiers, but also have a notion of time. So, inside process, you'd call metrics.store(String resourceIdentifier, T data) and the Metrics class would then store that data as a time series. If you need to clear some metric, you'd do so with the metrics.clean(String resourceIdentifier), optionally passing options for what to clear (old data? new data?). The Metrics class could also allow metrics.aggregate(String resourceIdentifier) which would maybe take a strategy for aggregation as a parameter.

So your example would look more like this, implementing the above design:

class Supervisor {
    private:
        // collector, metrics
    public:
        void process() {
            for (T resource : collector.resources()) {
                // process resource
                metrics.store(resource.getIdentifier(), resource.getData());
            }
        }
};

Collector collector();
collector.collect("resource1", resource1); // maybe overload with options for configuration?
collector.collect("resource2", resource2);

Metrics metrics();

Supervisor supervisor(collector, metrics);

while (condition) {
    supervisor.process();

    T output metrics.get("resource1");
}

Of course, using the composition pattern like this, you could have Supervisor hide the fact that it is a composition, so you could implement the Supervisor.getMetric(String resourceIdentifier) T method, which would of course just pass the call on to the metrics member.

If, for your users, you don't want them to interact with Collector and Metrics at all, you could let Supervisor instantiate these members on its own, but allowing your users to utilize the Collector and Metrics contracts, they could inject their own subclasses.

The main takeaway from this answer should be that you should not fear capturing behaviour and state inside some class, if it logically can be separated from other behaviour and other state. So instead of your flat, one-class system, I've introduced three classes, but seeing that they logically can be separated, I do not think the API becomes over engineered or cumbersome to use.

Answer 2

There are a lot of ways to go about this from a distributed perspective e.g. push each atomic datum to a Kafka topic and have one or more aggregators compile the data. But based on the last requirement: "The user of the main class should have access to the metrics all the time", I think you are looking for an in-memory solution. I recently created something which I think is along the lines of what you are looking for here.

The basic idea is that you define some data points. These might be counts, time durations, or even more complex types that capture a single data point. So for the sake of argument, let's say you want to capture the latency for getting data from each resource. You could have a class like so:

class Latency {
  public static Latency start() {
    return new Latency();
  }

  public static class Duration {
    private final long start;
    private final long end;

    private Duration(long start, long end) {
      this.start = start;
      this.end = end;
    }

    public long getStart() {
      return start;
    }

    public long getEnd() {
      return start;
    }

    public long getDuration() {
      return duration;
    }
  }

  private final long start;

  private Latency() {
    start = System.nanoTime();
  }

  public Duration end() {
    return Duration(start, System.nanoTime());
  }
}

Probably overkill for this but the idea is that you can make this more or less sophisticated depending on your needs. Now can use this to track the time it takes to get a resource like so:

You need to store these durations somewhere. One thing that is important is to make sure that you can align these. You want to be able to associate a given latency for resource 1 with the corresponding one for resource 2. There are a number of ways to do this using collection indexes or looking at them as time series but the most robust way to do it is to have another type that holds a set of aligned statistics. For example you could use a Map<String, Duration> or create a custom type if you need something more fancy.

Now, you could just keep all of these Maps in a List and use them to calculate stats on the fly but as your application runs, you will continually use more and more memory. That might be OK but if not, then you probably want to aggregate these stats over time. It sounds like that's the idea you have. So you can throw these Maps into a queue or similar structure. Then on your schedule, you clear the queue and calculate your aggregate stats e.g. count, min max, median, mean, ... This probably means another type or some sort of dictionary/map to hold the aggregate data. You'd also want to capture the time-frame for each aggregate.

Lastly, if your application is long-running, you may need to age-off the aggregates to avoid (again) using too much memory, albeit at a slower pace. Another option would be to continually accumulate into a single set of aggregate statistics depending on the goal.