Web API: REST-ish pattern for flexible querying of time-series data

Question

I'm working on a product that exposes a RESTful API to interact with system entities (the usual CRUD), which works very well.

In addition to CRUD, we've recently encountered a need to expose the ability to query different time series based data, with different filters and at varying aggregation resolutions (but based on pretty much the same aggregate functions).

For example, consider a system that manages wind turbines. You would have usual CRUD operations to add / remove turbines, but also some kind of API to query the turbines electrical output or the rotation speed over time. You might want to get these metrics averaged out across all turbines in a site, or for a single turbine, or for all turbines from a specific maker (these are your "filters"). You might want the raw data (that is every measurement done in a given time period), or hourly data points, or daily or weekly ones (this is your granularity or resolution).

You can safely assume that all queries have a start / end time filters and that sorting is always based on time.

The main consumer of this data is a GUI that renders charts based on it. It is helpful, although not strictly required, that multiple metrics are encapsulated in a single endpoint (that is you can get both the turbine's rotation speed and output for a point in time with a single query).

What we've done historically is consider such a data point as a REST entity (say we call it TurbineStats), and have an endpoint that only supported listing entities, and accept filters & resolution as query parameters. For example:

GET /turbine/stats ⏎
    ?start=2018-01-01T10:11:12 ⏎
    &end=2018-01-01T10:11:12 ⏎
    &site_id=1234abcd ⏎
    &model=ABCXYZ ⏎
    &resolution=hourly

[
    {     
         "timestamp": "2018-01-01T11:00:00",
         "rotation_speed": 523.53,
         "output_kwh": 20.5
    },
    {     
         "timestamp": "2018-01-01T12:00:00",
         "rotation_speed": 433.13,
         "output_kwh": 18.5
    },
    {     
         "timestamp": "2018-01-01T13:00:00",
         "rotation_speed": 569.0,
         "output_kwh": 24.1
    },
    ...
]

This works, but feels to us more and more like an anti-pattern, mostly because:

• While entities look more or less the same and have the same fields, the data can have different meaning depending on the query. For example rotation_speed can be exact or averaged.
• There's important meta-data (like what are the filters, how many points were considered, how many turbines are included, etc.) which can be included as part of the entities but mixing data and metadata feels awkward.
• Requesting a single entity makes little sense. This is basically a "list only" API
• Entities are computed and do not have unique identifiers. We can come up with computed identifier but it feels like we're doing that just to be "RESTful" and not for a real use.

Can you give an example of time series querying API that works well and feels well-designed, RESTful or not? Are there any good patterns to look at?

Answer 1

REST is not perfect. It's essentially object focused, and you create and delete those objects. That means OPERATIONS are not in the REST sweet spot ;-)

Before REST became popular, the 'in' way to do web-services was SOAP. It had no objects, and instead was OPERATION focused. It handled beautifully what you are looking for.

The truth is that neither approach works very well, but REST is in vogue today, so it makes sense to structure your web services using this approach.

But start out understanding, that for a good percentage of your design (how much vary's a great deal from project to project) - you will need to stretch the definitions/best practices to make sense for your application.

What many people do with REST to deal with this problem, is conceptualize 'virtual objects'. These (generally) cannot be created, but are in essence views or ways to package up your operations in vaguely a object-oriented organization.

That's basically what you've come up with on your own (except that when people do it, they often make the objects sibling to the CRUD objects, not sub-objects), its the best approach I'm aware of when using a REST pattern.

Answer 2

Check the Prometheus API:

https://prometheus.io/docs/prometheus/latest/querying/api/#expression-query-result-formats

They designed their query-endpoint to accept GET (for simple queries) and POST (for large queries that may breach server-side URL character limits) requests.