Designing CRUD part of an HTTP API

Question

This question is in the context of web based applications. A web server exposing an HTTP api for clients (e.g. running in a browser but not necessarily). Usually the web server would be connected to somekind of datastorage.

My concern is the API exposed by the HTTP server, and how to design it.

I find that, often, a big part of api exposed to the client is just a CRUD interface to the datastorage. For example, the api almost always contains things like:

- get_all_users()
- get_this_user(id)
- create_new_user(name, email, phone, password)
- update_user(id, name?, email?, phone?, password?)
- delete_user(id)

This is not all the API does, but it something that is common to a lot of APIs I've worked with.

Depending on what the client needs, I often end up with more and more things in the API such as these:

- get_user_by_email(email)
- get_all_users_along_with_number_of_lincenses_for_each()
- delete_all_these_users_at_once(array [])
- get_users_whose_profile_is(profile_name)
- get_names_of_users_who_are_admin_profiles()
- .... etc

In other words, the client often needs to do joins, filter, and sort the data in all kinds of ways.

Which has led me to overly bloated APIs (just like the one above) supporting all kinds of options for filtering, sorting, and joining with data from other tables.

The resulting API is too complex for both the client to use and learn and for the server to maintain and test.

(IMO the Zabbix HTTP api suffers from this syndrom).

Questions:

• Do you have similar experience?

• Do you have some pointers as to how to strike a good balance between API completeness and complexity.

PS. I'm aware of REST, but I don't find it resolves the problem, the api provided is still very basic, and doesn't solve the get_users_along_with_number_of_licenses() or the get_names_of_users_which_are_admin() api calls.;

Answer 1

This is a classic API design dilemma, regardless of whether it's provided by a web service, or by linking a library, or by just being part of a code base that gets built along with everything else. It boils down to whether you prefer many functions that each do one specific thing, or whether you have fewer, more parameterized functions.

Since web calls are more "expensive" in terms of time, many web API designers opt for the fewer, richer, calls. A CRUD interface over HTTP would provide Read (GET), Write/Create (PUT/POST), and Delete (DELETE) operations, with all necessary parameters being provided in the request body or encoded as parameters in the URL (QueryString).

To address your ever-growing list of ways to filter and organize your Read operations, you should define a richer set of inputs, which can be as complex and structured as you want, since HTTP will deliver anything you ask it to. Most folks design their parameters as name/value pairs, or as Json objects, or as xml documents. Your parameters can include filtering criteria, and options indicating what data you would like the call to return.

Atlassian does this with their REST APIs for JIRA and Confluence (and others). They provide URLs (calls) to get all issues, or one issue, or some issues based on a filter or query. Each GET call supports a parameter (in this case, called expand, which allows the caller to ask that certain optional return data be included in the response.

As for sorting, if your service will be returning lots of data, or chunked (paged) data, it is nice of you design the call with a sort parameter to make life simpler for the caller. For smaller lists of data, it might be easier to just provide the data in one ordering, and let the client sort it however they like.

REST is currently popular because it is highly decoupled between caller and provider, which offers extreme flexibility. You may prefer something like XMLRPC, which looks more like function calls to client code. You can still provide a call that accepts many parameters, or one large object-like parameter. It's still up to you.

Do yourself, and your users a favor, and make sure you document your API thoroughly, and make it readily available. The trend in web services lately is to move away from tight bindings (as in SOAP and XMLRPC), which makes it harder for compilers and IDEs to help you know how to make a call. Thorough, accessible documentation will be your friend.

Answer 2

I have done such things as REST, which, as you say, doesn't solve all problems, though at least a few of them possibly.

Basically search features I have implemented using URL params instead having a single URL for each of them. So your get_user_by_email would just be

/user?email=joe.doe@something.net

Similar you can handle the returned data for things like get_all_users_along_with_number_of_lincenses_for_each

/users?fields=number_licenses

Or even return nested data like

/users?includes=licenses

Can even be several includes:

/users?includes=licenses,orders

which would return a nested JSON object with users and all the license records.

or for your get_names_of_users_who_are_admin_profiles like this:

/users?admin=true&fields=id,name

The method get_users_whose_profile_is would be a simple nested route:

/profiles/{profile_id}/users

Of course this doesn't fully solve the complexity problem, but at least reduces the amount of endpoints the user has to access and learn. Still he needs to know which fields exist and how to query them (we even have some full text search features for some resources).

Another problem here could be the permissions, since some users may be allowed to search and request different types of data. Especially which fields he is allowed to request can be problematic. Still at least with the reduced endpoints the first level of permissions is easier (user can request /users/ or he can not.

How easy this is to implement may depend on your technology stack. We use Ruby on Rails that allows to write a few generic methods that make it easy to process things like fields=name,id,street,city or includes=orders,licenses (though be careful, this again needs to ensure that permissions are handled properly)

Answer 3

I suggest that you 1) learn what other REST APIs do, and 2) consider OData

I love Odata because it supports every scenario you listed in one single API call. I'm biased though, because in C# you can implement an Odata server in just a few lines of code if you use the Entity Framework. There's also client side libraries that can format OData URLs.

Let me apply some of your query examples to OData:

• get_user_by_email(email)

http://server/Users$filter=email eq 'joe@company.com'

• get_all_users_along_with_number_of_lincenses_for_each()

http://server/Users$filter=email eq 'joe@company.com'/Licenses

• get_names_of_users_who_are_admin_profiles()

http://server/Users$filter=admin eq true$select=Name

Overall, OData basically maps everything you would find in a SQL select clause into the Query string. The only complaint I have about it is that it is a bit too powerful so you need to do some work on the server side to make sure someone doesn't do a complicated query for a million records. Limit the number of results, limit the time the query can run, etc.

With your delete example, there are a some common ways to pass arrays. If you read the documentation on Swagger you can get an idea of the different ways that people tend to pass arrays as arguments. Swagger lists comma separated values, space separated, tab separated, pipe separated, and repeating the parameter multiple times. CSV is the default, so that tells me that people tend to use it a lot. ASP.NET MVC natively maps the last example to an array. So consider:

• delete_all_these_users_at_once(array [])

HTTP DELETE http://server/Users?id=7&id=14&id=8375 or HTTP DELETE http://server/Users?id=7,14,8375