5

Let's say we have schools with some data including a name and a list of students, and students with some data including courses they're enrolled in and a reference to their school. On the client:

  • I'd like to show a screen that shows information about a school, which includes a list of all of its students by name.
  • I'd like to show a screen that shows information about a student, including the name of their school and the names of courses they're taking.
  • I'd like to cache this information so that I can show the same screen without waiting on a new fetch. I should be able to go from school to student and back to school without fetching the school again.
  • I'd like to show each screen with only one fetch. Going from the school page to the student page can take a separate fetch, but I should be able to show a school with the full list of student names in one fetch.
  • I'd like to avoid duplicating data, so that if the school name changes, one fetch to update the school will lead to the correct name being shown both on the school page and the student pages.

Is there a good way to do all of this, or will some of the constraints have to be lifted?

A first approach would be to have an API that does something like this:

GET /school/1

{
    id: 1,
    name: "Jefferson High",
    students: [
        {
             id: 1
             name: "Joel Kim"
        },
        {
             id: 2,
             name: "Chris Green"
        }
        ...
    ]
}


GET /student/1

{
    id: 1,
    name: "Joel Kim",
    school: {
        id: 1,
        name: "Jefferson High"
    }
    courses: [
        {
             id: 3
             name: "Algebra 1"
        },
        {
             id: 5,
             name: "World History"
        }
        ...
    ]
}

An advantage of this approach is that, for each screen, we can just do a single fetch. On the client side, we could normalize schools and students so that they reference eachother with IDs, and then store the objects in different data stores. However, the student object nested inside of school isn't a full object -- it doesn't include the nested courses, or a reference back to the school. Likewise, the school object inside of student doesn't have a list of all attending students. Storing partial representations of objects in data stores would lead to a bunch of complicated logic on the client side.

Instead of normalizing these objects, we could store schools and students with their nested partial objects. However, this means data duplication -- each student at Jefferson High would have the name of the school nested. If the school name changed just before doing a fetch for a specific student, then we'd show the right school name for that student but the wrong name everywhere else, including on the "school details" page.

Another approach could be to design the API to just return the ids of nested objects:

GET /school/1

{
    id: 1,
    name: "Jefferson High",
    students: [1, 2]
}


GET /student/1

{
    id: 1,
    name: "Joel Kim",
    school: 1,
    courses: [3, 5]
}

We'd always have "complete" representations of objects with all of their references, so it's pretty easy to store this information in data-stores client side. However, this would require multiple fetches to show each screen. To show information about a student, we'd have to fetch the student and then fetch their school, as well as their courses.

Is there a smarter approach that would allow us to cache just one copy of each object, and to prevent multiple fetches to show basic screens?

  • 3
    Your problem has less to do with REST than it has with your actual application architecture. – Andy Aug 7 '16 at 21:40
  • @DavidPacker I think you're right here. I think there may be ways to communicate the data over the API that will make architecting a solution easier. But in the end, figuring out how to deal with storing and updating "partial" objects coming over the wire in a way that minimizes data inconsistency is the goal. – Michael Marvick Aug 7 '16 at 21:51
  • 1
    @MichaelMarvick I think Luc Franken has a brilliant answer below. For a good way of organizing your backend architecture, you can have a look at CQRS. It suits that idea of "denormalizing" pretty well. – guillaume31 Aug 8 '16 at 10:11
5

In REST you can see everything you listed as a different resource. It is a different approach. In general you want to denormalize (database term) the data in a REST resource to fit your use case. So your examples are just fine, it includes all data you need at once so you can easily build a client for that.

In REST we can say you have the following resources: * /schools (list of schools) * /schools/1 (school with list of students) * /students (maybe list of students) * /students/1 (student with list of courses for example). * /courses (list of all courses) * /courses/1 (list of one course which may contain all students enrolled)

You could also have in addition: * /schools/1/students (list of all students at this school) * /students/1/courses (all courses for this student) * /students/1/courses-finished (all finished courses for this student)

Is there a smarter approach that would allow us to cache just one copy of each object, and to prevent multiple fetches to show basic screens?

Object != resource in REST. The list of schools for example is also a resource. So you might have 3 database tables here but multiple resources.

Client-side data stores Having data stored in separate datastores on the client is a different matter. I know it has been implemented in Meteor: https://meteorhacks.com/understanding-mergebox/ which allows to have an initial smaller document which expands when needed.

Subscriptions So what you do is: You subscribe to schools and students (only info needed for listings, so for example: id & name). Then when you know which student you want more details off you subscribe to that student in full detail. Behind the scenes it will populate your client-side database with all details of that specific student.

REST That's not about REST, it's a totally different way of working. With REST, in general, you should not have to associate resources on the client.

Normalisation

Instead of normalizing these objects, we could store schools and students with their nested partial objects. However, this means data duplication -- each student at Jefferson High would have the name of the school nested.

No. A REST resource (school/student/etc) is NOT equal to your database or whatsoever. (http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven#comment-743) You can return the REST resource school by making a join query behind the scenes. REST does and should not care about that. If performance becomes an issue you can just de-normalize that query behind the scenes and the REST resource is still looking the same. If you switch databases, resource looks still the same. Switching to another students management system, resource still the same. That's a big advantage you get by using REST. Originally it was designed to connect lots of unknown systems, by creating a universal API not linked to the backend technology you can work better together.

Caching When a school name changes your should invalidate all caches using that school name. You can do that using ETAG. A quite simple example to understand this is: http://fideloper.com/api-etag-conditional-get

So, let's assume you use a join to generate the school resource with the list of student names. And let's assume that you have a modifiedAt field in both schools and students table:

When you include both modifiedAt fields in your resource (or at least use them) and you generate a unique ETAG the following will happen:

  • When school name is updated: ETAG becomes invalid for ONE school and ALL students of that school.

  • When student name is update: ETAG becomes invalid from ALL schools where the student is located and ONE student (the one changing).

That allows the caches in between all to know when to invalidate their cache. You can also pre-cache those resources. ETAG allows to return the cached instance. So if the user would be offline he could just use the local resources. You could for example after login load all resources by requesting them in the background for this user:

/students/1
/student/1/courses (even all filtered courses)
And then all courses he is enrolled into.

They will then be in the local cache and are available. This is also used for optimistic loading. You could just show the on from the cache instantly. Then in the background check to see if they are updated. If they are reload the data on screen.

Understand that there are multiple caches: The server may use one, in between some cache might be active, the browser caches, your own code may cache. This universal way gets almost all well done HTTP caches to behave like you want.

HATEOAS Read about HATEOS which makes it easier for you to fetch those students. Just include an url to them in the school resource. Then you can just follow the links. Examples: https://www.infoq.com/articles/webber-rest-workflow and an example: https://developer.paypal.com/docs/integration/direct/paypal-rest-payment-hateoas-links/

That makes it more manageable to get those sub resources.

It will allow you for example to show a list view with links on the student page to all his courses. And on top you might show a link to all his finished courses (when available in the api). And that business logic would all be handled by the api. So your client could just do:

/students/1 (so you get all data on the student)

if(student.links.finished) {
  console.log(student.links.finished);
}

When he has no finished courses you could hide the link that way for example.

This can also be expanded like:

student.links.courses {
  latest: /students/1/new-courses
  finished: /students/1/courses-finished
  failed: /students/1/courses-failed
}

That way you could on your client just do a forEach loop and show all available filters.

Strategy Off course you can mix both. You could write a merge box implementation like which uses a REST api. That might be worth it if you want to re-use that data on the client and do for example complex searches / filtering on them. That's the advantage of having a client-side database.

On the other hand: If you just deliver the right data as REST resources there is no direct advantage of converting that data into a database at client side and then read from that. If you could just use the original resource because it contains just what you need.

So it is a tradeoff depending on your exact use-case. If you can keep it simple and just use the REST resources that will work just fine. And it's simple which is a great advantage.

If you have to do complex data searches etc. the local client database might be a good fit.

  • This was a huge help. Your point that objects are not rest resources helped me understand this better, and it seems like using ETAGs to invalidate things when appropriate is a good approach for us. We aren't doing any sort of complex filtering or data manipulation, so just returning the right data as a REST resource seems like the way to go. Thank you for the detailed response! – Michael Marvick Aug 9 '16 at 7:31
  • Great to hear it's of use for you Michael! Good luck with your project! – Luc Franken Aug 9 '16 at 7:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.