Web Service API Return Type

Question

I'm trying to find some opinions on best practice for returning different types of data for the "same" endpoint. Surprisingly, I can't find much on this subject.

Basically, I am building an API for our website, which is a single-page app. The client uses caching to store data across pages and this data is updated as necessary by the server (using websockets). There are several routes that return a list of data. Each item in the list can potentially be quite large and the list itself can contain many items.

Since the frontend already uses caching, we want to reduce the amount of unnecessary data resent to the client where possible. One option to accomplish this is to provide a way for the frontend to obtain a list of the IDs of resources for a particular request. Based on this list, the client could then request the data it needs (i.e. if a cache miss occurs).

This approach is interesting, but it raises several questions.

• What if the frontend already knows it doesn't have any of the resources? Instead of requesting a list of IDs, it should be able to request the data directly.
• What is the best approach? Possible approaches and their (dis)advantages appear below.

Question

Is there an approach not listed below that seems appropriate? Is there an approach I missed that might be better? Does this seem like overkill?

Aproaches

1. Just send the data - Nuclear option #1.

   Advantages

   • Simplicity
   • Return type is consistent

   Disadvantages

   • The client may already have much of the data, wasting bandwidth

2. Use the same route and declare a query param that determines the type of data to return - While not the worse option, it does seem dirty returning different types of data from the same endpoint (yes, I know you can specify Content-Type in requests, but this header is telling the server what representation of the data is desired, not really what type of data to return).

   Advantages

   • The client can determine when it wants the data or just the IDs

   Disadvantages

   • Return type depends on the value of the query param
   • Added conditional logic

3. Allow for field selection. - This approach follows Suggestion 7 on this blog.

   Advantages

   Disadvantages

   • Added unnecessary complexity - Field selection doesn't make sense for our app. Therefore, the only valid value for this param would be the ID field
   • The format of the data depends on the query param - while the data type is technically still the same, its structure is different in each request. Not something I really like.

4. Prepend the original route with /ids - Clients would be able to send a GET request to the same path they would for the actual data prepended by /ids. For example, instead of /cars, the request would be sent to /ids/cars.

   Advantages

   • Seems more semantically meaningful than just using a query param

   Disadvantages

   • GET would be the only logical request that would ever be sent to these endpoints
   • DRY or added conditional logic - Either a new endpoint has to be added that basically duplicates the original endpoint or the same endpoint is used with the same conditional logic necessary for point 2

5. Use a value for Accept and Content-Type headers similar to application/json+id - This is an approach similar to the one used by LTI. The advantages and disadvantages (as far as I know) are the same as point 2.

6. Always send just the ID - Nuclear option #2.

   Advantages

   • Simplicity
   • Return type is consistent
   • Potentially results in less data being sent to the client

   Disadvantages

   • Since the number of resources available could be high, this could result in many GET requests (200+) if the cache is (almost) empty, ultimately defeating the purpose of this approach
   • Semantically less meaningful - I'm making a request for the resources, not their IDs!

Answer 1

For something like this, an approach I've used on a project that works fairly well is to use something like Falcor. At a very high level, Falcor provides the following:

• A unified graph API to request the specific data you intend to display
• Centralized caching
• Optimized server queries (i.e. it only requests what is missing locally)

To be clear, there are other graph API frameworks out there, but Falcor is the one I have experience with.

Falcor is split between a client side component and a server side component.

• The client side caches the local information and determines what deltas to request
• The server side has "routes" to pull data from full API servers--or implement everything directly
• The server side can invalidate cache entries which helps the UI stay fresh

All that said, based on my experience is that you don't get flexibility like that without some cost. That cost comes in the form of a steep learning curve to grasp exactly what it is doing for you. The fact that it has a UI layer tier and a server layer tier changes how you design your application. Essentially, instead of managing your caches yourself, you trust the tool to do it for you.

I'm positive I've only scratched the surface, and this is only one of the ways to deal with what you are describing. It has a fairly good integration with React so that the different components on screen can be updated as needed. I.e. if the server invalidates an entry that is in use elsewhere on screen, Falcor is smart enough to package up the missing data with the next request.

Pros

• It combines Options 2 and 3 in your question
• It handles caching on your behalf
• Your UI only gets the data it needs for display

Cons

• It has a steep learning curve (as does other tools like it)
• The graph query interface is not very intuitive
• It adds another layer to your app

I'll be honest, I wouldn't use it for a simple application. However if you are incorporating elements from several microservices on screen at once in a SPA, it can be a very useful tool to minimize the traffic between the client and server while also keeping the UI as a whole up to date.

Answer 2

There's a lot here and I probably won't cover every bit of what you mention in the question. Basically you have a bunch of resources and you don't want to reload them all every time. I worked on something like this about 12 years ago where we had a mega-document (here's everything) interface that we pulled down from the server. It was slow (more on this later) for normal users but we had people on DSL lines who where trying to load one response took something crazy like 15 minutes. We didn't want to redesign the entire solution I created a local client cache that held all shared data. The client would then check each type of resource (we didn't call it that but it's the correct term) and see if it had been updated since it was loaded to the client. This reduced the amount of data being loaded and made the experience tolerable enough that we didn't get fired.

First off, I don't think there's any reason to avoid returning different representations of the same data even some are less detailed than others. The falcor graph is, in my mind a great candidate for a registered media type but it doesn't seem to be one. No major matter, though, you can always use the unregistered x tree. Using the Accept header is the proper way to do this. That's why media types exist.

The design you seem to want to go with is that in the case that you have nothing, you want to pull it all down in one shot. If you have a lot of data, as you state, you really probably don't want to do that. This will perform much worse than getting the list of resources and pulling each one. The main reason is that, in general, you first make this giant message on the server, then you need to pull down every single byte of that monster, and only then can you start doing anything with the data. This not only means a big delay, it also will stress the client as it needs to hold all of that in memory. If you are intent on this route, look at a streaming JSON parser like OboeJS.

What I would do, if I were you is create a graph using the JSON graph spec from Falcor (why not?) and attach the last modification timestamp to each resource you list in the graph. Then simply walk the tree breadth first and retrieve anything that you don't have or is newer than what you have. This works well if the structure of your tree is shaped like the structure of your UI workflow. For example, say your initial page shows a list of documents. Make sure you pull down the list of documents first. Then go back and get the content (if you wish to prefetch) of those documents. If you are lucky, you'll pull it down before they click on it or you will already be in the process of downloading it. If not, move the priority of that item up in your retrieval. Since you just need to pull the one thing, if shouldn't take long anyway.

The problems I had over a decade ago were all a result of trying to pull down everything in a big ball of mud. The next system was designed to pull each thing down separately. I was told by smart people that it would be slower or have too much overhead. The results were unambiguously that the mud ball is a bad idea. The clients hated the mud ball. They loved the design that pulled each piece. The reason was that had to wait for the mud ball. The hierarchical design seemed mostly instantaneous. They weren't aware of any delays in getting data from the server.