3

I'm working on a product that exposes a RESTful API to interact with system entities (the usual CRUD), which works very well.

In addition to CRUD, we've recently encountered a need to expose the ability to query different time series based data, with different filters and at varying aggregation resolutions (but based on pretty much the same aggregate functions).

For example, consider a system that manages wind turbines. You would have usual CRUD operations to add / remove turbines, but also some kind of API to query the turbines electrical output or the rotation speed over time. You might want to get these metrics averaged out across all turbines in a site, or for a single turbine, or for all turbines from a specific maker (these are your "filters"). You might want the raw data (that is every measurement done in a given time period), or hourly data points, or daily or weekly ones (this is your granularity or resolution).

You can safely assume that all queries have a start / end time filters and that sorting is always based on time.

The main consumer of this data is a GUI that renders charts based on it. It is helpful, although not strictly required, that multiple metrics are encapsulated in a single endpoint (that is you can get both the turbine's rotation speed and output for a point in time with a single query).

What we've done historically is consider such a data point as a REST entity (say we call it TurbineStats), and have an endpoint that only supported listing entities, and accept filters & resolution as query parameters. For example:

GET /turbine/stats ⏎
    ?start=2018-01-01T10:11:12 ⏎
    &end=2018-01-01T10:11:12 ⏎
    &site_id=1234abcd ⏎
    &model=ABCXYZ ⏎
    &resolution=hourly

[
    {     
         "timestamp": "2018-01-01T11:00:00",
         "rotation_speed": 523.53,
         "output_kwh": 20.5
    },
    {     
         "timestamp": "2018-01-01T12:00:00",
         "rotation_speed": 433.13,
         "output_kwh": 18.5
    },
    {     
         "timestamp": "2018-01-01T13:00:00",
         "rotation_speed": 569.0,
         "output_kwh": 24.1
    },
    ...
]

This works, but feels to us more and more like an anti-pattern, mostly because:

  • While entities look more or less the same and have the same fields, the data can have different meaning depending on the query. For example rotation_speed can be exact or averaged.
  • There's important meta-data (like what are the filters, how many points were considered, how many turbines are included, etc.) which can be included as part of the entities but mixing data and metadata feels awkward.
  • Requesting a single entity makes little sense. This is basically a "list only" API
  • Entities are computed and do not have unique identifiers. We can come up with computed identifier but it feels like we're doing that just to be "RESTful" and not for a real use.

Can you give an example of time series querying API that works well and feels well-designed, RESTful or not? Are there any good patterns to look at?

  • 1
    what about the graphite api? – Ewan Jul 24 '18 at 15:06
  • @Ewan link / example? – shevron Jul 24 '18 at 15:11
  • Thanks. Good suggestion (hence the upvote) but I feel this pattern goes full RPC and shifts all the flexibility to the query= parameter, where as in my case there is a lot of defined domain boundaries and I can be less flexible and more explicit about endpoints and parameters. – shevron Jul 24 '18 at 15:20
  • 1
    Regarding the last two bullets: As I see it, this endpoint returns a single entity which happens to contain a list of datapoints. I don't see why you would need to treat the datapoints as entities of their own, unless you want to be able to retrieve them individually. – Bart van Ingen Schenau Jul 25 '18 at 7:33
1

REST is not perfect. It's essentially object focused, and you create and delete those objects. That means OPERATIONS are not in the REST sweet spot ;-)

Before REST became popular, the 'in' way to do web-services was SOAP. It had no objects, and instead was OPERATION focused. It handled beautifully what you are looking for.

The truth is that neither approach works very well, but REST is in vogue today, so it makes sense to structure your web services using this approach.

But start out understanding, that for a good percentage of your design (how much vary's a great deal from project to project) - you will need to stretch the definitions/best practices to make sense for your application.

What many people do with REST to deal with this problem, is conceptualize 'virtual objects'. These (generally) cannot be created, but are in essence views or ways to package up your operations in vaguely a object-oriented organization.

That's basically what you've come up with on your own (except that when people do it, they often make the objects sibling to the CRUD objects, not sub-objects), its the best approach I'm aware of when using a REST pattern.

  • 1
    The problem with REST appears to be a lack of imagination. Nothing prevents anyone from simply creating the data structures that are most suitable for their application and using POST. – Robert Harvey Jul 24 '18 at 14:54
  • I haven't said the design for these query endpoints has to be REST or RESTful. It's an option but not a must. I've been around long enough to do SOAP and other RPC paradigms. For most of what we do, REST is just right but for query endpoints we think it's not. I'm specifically asking if there are well known or tried out patterns for solving this case. I can of course come up with my own pattern(s), but that's not the point. – shevron Jul 24 '18 at 15:04
  • @RobertHarvey this is not a REST vs. whatever fight. Doesn't have to be REST, but I am asking about a common pattern to follow. Yes, I can "imagine" my own pattern, but following a familiar pattern, assuming it works, is actually a good thing. – shevron Jul 24 '18 at 15:06
  • Also @RobertHarvey why would I use POST? I'm reading data, and there are no limits here that using POST over GET would solve. The place where I think common REST patterns fall short here is more about the response entity structure than the request method / parameters. – shevron Jul 24 '18 at 15:08
  • The thing about fads, is that when they are hot, there are always (even smart people) who instinctively feel that if you can just stick to the purity of the paradigm, you can solve all problems. While it MIGHT POSSIBLY be true - what you @RobertHarvey - are implying about it being POSSIBLE to design a REST API such that there is no need for 'virtual objects' (actually it isn't), it misses the point. The attractiveness of REST lies principally in the fact that it facilities building a MODEL based on INTUITIVE objects from a given domain. That pragmatically constrains modeling choices. – Lewis Pringle Jul 24 '18 at 15:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.