136

When designing a RESTful interface, the semantics of the request types are deemed vital to the design.

  • GET - List collection or retrieve element
  • PUT - Replace collection or element
  • POST - Create collection or element
  • DELETE - Well, erm, delete collection or element

However, this doesn't seem to cover the concept of "search".

E.g. in designing a suite of web services that support a Job Search site you might have the following requirements:

  • Get individual Job Advert
    • GET to domain/Job/{id}/
  • Create Job Advert
    • POST to domain/Job/
  • Update Job Advert
    • PUT to domain/Job/
  • Delete Job Advert
    • DELETE to domain/Job/

"Get All Jobs" is also simple:

  • GET to domain/Jobs/

However, how does the job "search" fall into this structure?

You could claim it's a variation of "list collection" and implement as:

  • GET to domain/Jobs/

However, searches can be complex and it's entirely possible to produce a search that generates a long GET string. That is, referencing a SO question here, there are issues using GET strings longer than about 2000 characters.

An example might be in a faceted search - continuing the "job" example.

I may allow for searching on facets - "Technology", "Job Title", "Discipline" as well as free-text keywords, age of job, location and salary.

With a fluid user interface and a large number of technologies and job titles, it is feasible that a search could encompass a large number of facet choices.

Tweak this example to CVs, rather than jobs, bring in even more facets, and you can very easily imagine a search with a hundred facets selected, or even just 40 facets each of which are 50 characters long (e.g. Job Titles, University Names, Employer Names).

In that situation it might be desirable to move a PUT or POST in order to ensure that the search data will get correctly sent. E.g.:

  • POST to domain/Jobs/

But semantically that's an instruction to create a collection.

You could also say you'll express this as the creation of a search:

  • POST to domain/Jobs/Search/

or (as suggested by burninggramma below)

  • POST to domain/JobSearch/

Semantically it may seem to make sense, but you're not actually creating anything, you're making a request for data.

So, semantically it's a GET, but GET isn't guaranteed to support what you need.

So, the question is - Trying to keep as true to RESTful design as possible, whilst ensuring that I'm keeping within the limitations of HTTP, what is the most appropriate design for a search?

  • 3
    I often intent to use GET domain/Jobs?keyword={keyword}. This works fine for me :) My hope is, that the SEARCH verb will become a standard. programmers.stackexchange.com/questions/233158/… – Knerd Mar 21 '14 at 11:05
  • Yes, I can see that for a trivial example there isn't a problem. But in the tool we're building it's actually not that unbelievable that we'd end up with a complex search that results in a GET string longer than 2000 characters. What then? – Rob Baillie Mar 21 '14 at 11:13
  • Actually a very good point. What about specifing a compression technology? – Knerd Mar 21 '14 at 11:16
  • 2
    GET with a body is allowed by the HTTP spec, may or may not be supported by middleware (sometimes not) ;) and isn't favored as a practice. This comes up on Stackexchange periodically. stackoverflow.com/questions/978061/http-get-with-request-body – Rob Mar 21 '14 at 15:17
  • 2
    I ended up having POST JobSearch create an actual search entity and returning a jobSearchId. Then GET jobs?jobSearch=jobSearchId returns the actual jobs collection. – Cerad Mar 21 '14 at 15:46
92

You should not forget that GET requests have some superior advantages over other solutions:

1) GET requests can be copied from the URL bar, they are digested by search engines, they are "friendly". Where "friendly" means that normally a GET request should not modify anything inside your application (idempotent). This is the standard case for a search.

2) All of these concepts are very important not just from user and search engine, but from an architectural, API design standpoint.

3) If you create a workaround with POST/PUT you will have problems which you are not thinking of right now. For example in case of a browser the navigate back button / refresh page / history. These can be solved of course, but that's going to be another workaround, then another and another ...

Considering all this my advice would be:

a) You should be able to fit inside your GET with using clever parameter structure. In extreme case you can even go for tactics like this google search where I set a lot of parameters still its a super short url.

b) Create another entity in your application like JobSearch. Assuming you got so much options, its probable that you will need to store these searches as well and manage them, so its just clearing up your application. You can work with the JobSearch objects as a whole entity, meaning you can test it / use it easier.


Personally I would try to fight with all my claws to get it done with a) and when all hope is lost, I would crawl back with tears in my eyes to option b).

  • 4
    For clarification, this question is intended to be about web-services design, not web-site design. So whilst the browser behaviour is of interest in the wider scope of the question's interpretation, in the particular case described it's of no consequence. (interesting point though). – Rob Baillie Mar 21 '14 at 13:02
  • @RobBaillie Ye the browser was just a use case. I wanted to express the fact that your search as a whole is represented by a URL string. Which has a lot of comfort in usability along with other points later in the answer. – p1100i Mar 21 '14 at 13:23
  • In point b, is this a simple variation of my own reference to a POST to domain/Jobs/Search/, maybe with domain/JobsSearch/ instead, or did you mean something different? Can you clarify? – Rob Baillie Mar 21 '14 at 13:31
  • 7
    Why am I getting the impression that REST is quite often part of the problem, rather than part of the solution? – JensG Mar 27 '14 at 16:47
  • 1
    "GET request should not modify anything inside your application (idempotent)" while GET is idempotent, the relevant word is "safe" here. Idempotent means that doing GET on a resource twice is the same as doing GET on that resource once. PUT is also idempotent, but not safe, for example. – Jasmijn Feb 15 '17 at 3:07
12

TL;DR: GET for filtering, POST for searching

I make a distinction between filtering the results from listing a collection vs a complex search. The litmus test I use is basically if I need more than filtering (positive, negative, or range) I consider it a more complex search requiring POST.

This tends to be reinforced when thinking about what will be returned. I usually only use GET if a resource has a mostly complete life-cycle (PUT, DELETE, GET, collection GET). Typically in a collection GET I'm returning a list of URIs which are the REST resources that make up that collection. In a complex query I may be pulling from multiple resources in order to construct the response (think SQL join) so I won't be sending back URIs, but actual data. The problem is the data will not be represented in a resource, so I will always have to return data. This seems to me a clear cut case of requiring a POST.

--

It's been awhile and my original post was a bit sloppy so I thought I would update.

GET is the intuitive choice for returning most kinds of data, collections of REST resources, structured data of a resource, even singular payloads (images, docs, etc).

POST is the catchall method for anything that doesn't seem to fit under GET, PUT, DELETE, etc.

At this point I think simple searches, filtering do intuitively make sense through GET. Complex searches is up to personal preference especially if you are throwing in aggregation functions, cross correlations (joins), re-formatters, etc. I would argue the GET params should not get too long, and it a largish query (however it's being structured) can often make more sense as a POST request body.

I also consider the experience aspect of API usage. I generally want to make most of the methods as easy to use and intuitive as possible. I'll push calls that are more flexible (and therefore more complex) into POSTs and on a different resource URI, especially if it's inconsistent with the behavior of other REST resources in the same API.

Either way, consistency is probably more important than whether you are doing search in GET or POST.

Hope this helps.

  • 1
    Since REST is intended to abstract away the underlying implementation (e.g. - a resource is not necessarily a row in a database or a file on a hard drive, but could be anything) I don't know that it necessarily makes sense to use POST over GET when it comes to performing SQL joins. Suppose you have a table of schools and a table of children and you want a class (one school, multiple children). You could easily define a virtual resource and GET /class?queryParams. From the perspective of a user the "class" was always a thing and you didn't have to do any weird SQL joins. – stevendesu Aug 5 '15 at 17:17
  • There is no difference between "filtering" and "searching". – Nicholas Shanks May 1 '18 at 10:10
  • 1
    Yes there is, a filter is based on existing fields. A search may contain much more complex patterns, combining fields, computing adjecent values etc. – user13796 Jan 21 at 19:07
  • @stevendesu exactly, that's why I use POST for both (creating a search) :-) – ymajoros Jan 28 at 6:50
  • @ymajoros Unless you're saving the search terms and the results of the search somewhere, I don't know that POST makes sense semantically. When you perform a search you're making a request for information, you aren't providing new information to be retained anywhere. – stevendesu Jan 29 at 13:52
10

In REST, resource definition is very broad. It's really however you want to bundle some data.

  • It's useful to think of a search resource as a collection resource. The query parameters, sometimes called the searchable portion of the URI, narrow the resource down to the items the client is interested in.

For example, the main Google URI points to a collection resource of "links to every site on the Internet". Query parameters narrow that to the sites you want to see.

(URI = universal resource identifier, of which URL = universal resource locator, where the familiar "http://" is the default format for a URI. So URL is a locator, but in REST its good to generalize that to a resource identifier. People use them interchangably, though.)

  • Since the resource you're searching on in your example is the jobs collection, it makes sense to search with

GET site/jobs?type=blah&location=here&etc=etc

(return) {jobs: [{job: ...}]}

And then use POST, which is the append or process verb to add new items to that collection:

POST site/jobs

{job: ...}

  • Note that it's the same structure for the job object in each case. A client can GET a collection of jobs, using query params to narrow the search, and then use the same format for one of the items to POST a new job. Or it can take one of those items and PUT to its URI to update that one.

  • For really long or complicated query strings, convention makes it OK to send those as POST requests instead. Bundle the query paramters up as name/value pairs, or nested objects in a JSON or XML structure and send it in the body of the request. For example, if your query has nested data instead of a bunch of name/value pairs. The HTTP spec for POST describes it as the append or process verb. (If you want to sail a battleship through a loophole in REST, use POST.)

I'd use that as the fallback plan, though.

What you lose when you do that though is a) GET is nullipotent -- that is, it doesn't change anything -- POST is not. So if the call fails, middleware won't automatically retry or cache results, and 2) with the search parameters in the body, you can't cut and paste the URI anymore. That is, the URI isn't a specific identifier for the search you want.

To differntiate between "create" from "search". There are a couple of options that are consistent with REST practice:

  • You could do it in the URI by adding something to the name of the collection, like job-search instead of jobs. That just means you're treating the search collection as a separate resource.

  • Since the semantics of POST is both append OR process, you could identify search bodies with the payload. Like {job: ...} vs. {search: ...}. It's up to the POST logic to post or process it appropriately.

That's pretty much a design/implementation preference. I don't think there's a clear convention.

So, like you've already laid out, the idea is to define a collection resource for jobs

site/jobs

Search with GET + query params to narrow the search. Long or structured data queries go into the body of a POST (possibly to a separate search collection). Create with POST to append to the collection. And update with PUT to a specific URI.

(FWIW the style convention with URIs is to use all lowercase with words separated by hyphens. But that doesn't mean you have to do it that way.)

(Also, I should say that from your question, it's clear that you're a long way down this road. I spelled things out kind of explicitly just to line them up, but your question had already addressed most of the semantic issues in this answer. I was just lacing it with some convention and practice.)

  • It's an interesting idea - I wouldn't have considered using the payload to differentiate. It almost seems a little underhand! But I guess the URI scheme doesn't actually contain any verbs - it's the request type that defines the verb. Maybe the payload is closer semantically to the request type than the URI is. The only concern is - Is it transparent to a user of the API? – Rob Baillie Mar 21 '14 at 15:04
  • In terms of implementation (we're using Node and Express), it may mean the route can't really handle the choice of processing. I'd have to take a look at that... – Rob Baillie Mar 21 '14 at 15:06
  • I have the same gut feeling, that separating it by URI seems cleaner. I'm kind of going back and forth; it's a judgement call. However, the semantics of HTTP would allow putting it in the body. I like to say, REST is modeled after the World Wide Web, and the WWW was built with GET and POST. – Rob Mar 21 '14 at 15:11
8

I generally use OData queries, they operate as a GET call but allow you to restrict the properties which are returned and filter them.

You use tokens such as $select= and $filter= so you will end up with a URI which looks something like this:

/users?$select=Id,Name$filter=endswith(Name, 'Smith')

You can also do paging using $skip and $top and ordering.

For more information, check out OData.org. You haven't specified which language you are using, but if it's ASP.NET, the WebApi platform supports OData queries - for others (PHP etc) there are probably libraries you can use to translate them into database queries.

  • 6
    An interesting link and worth looking at, but does it solve the fundamental problem described, that GET requests do not support more than 2000 characters in the query string, and it is entirely possible that the query could be far longer than this? – Rob Baillie Mar 21 '14 at 12:28
  • @RobBaillie I don't think so as it is still a GET call with a query string. I'd suggest using OData wherever you can since it's a standard for querying web data sources and for the few (if any) times the query needs to be so complex that you can't fit it in a 2000 character query, create a specific endpoint which you make a GET call to – Trevor Pilley Mar 21 '14 at 13:19
  • Can you explain your approach for a "specific endpoint which you make a GET call to"? How might you imagine that endpoint would look? – Rob Baillie Mar 21 '14 at 13:33
  • @RobBaillie sure - again I'm not sure what tech you are using but in ASP.NET I'd create a specific controller called JobsNearMeAddedInTheLast7Days or whatever to encapsulate the query which is too long/complex for OData and then expose it only via GET calls. – Trevor Pilley Mar 21 '14 at 13:36
  • 1
    I see. Another interesting thought that probably has some legs, though I'm not sure this would help in my particular case - faceted searching with a lot of facet types and a lot of possible facet values – Rob Baillie Mar 21 '14 at 14:03
5

One approach to consider is treating the set of possible queries as a collection resource, e.g. /jobs/filters.

POST requests to this resource, with the query parameters in the body, will either create a new resource or identify an existing equivalent filter and return a URL containing its ID: /jobs/filters/12345.

The id can then be used in a GET request for jobs: /jobs?filter=12345. Subsequent GET requests on the filter resource will return the definition of the filter.

This approach has the advantage that it frees you from the query parameter format for filter definition, potentially providing you with more power to define complex filters. OR conditions are one example that I can think of that are difficult to accomplish with query strings.

A drawback to this approach is that you lose readability of the URL (although this can be mitigating by retrieving the definition though a GET request for the filter resource). For this reason you may also want to support the same or a subset of the query parameters on the /jobs resource as you would support for a filter resource. This could be used for shorter queries. If this feature is provided, in order to maintain cacheability between the two types of filtering, when using query parameters on the /jobs resource, the implementation should internally create/reuse a filter resource and return a 302 or 303 status indicating the URL in the form of /jobs?filter=12345.

  • My first reaction to this is that whilst it's good information, it's really just a variation on the answer provided by @burninggramma. Essentially it is "create a new entity called filter / search, call to create it and then call to retrieve it". The variation being that the call to retrieve it is more like a call to apply it to a collection. Interesting. However both your and burninggramma's answer suffer from the same problem - I have no desire to create the filters. There will be a huge number of them, and they don't need to be stored except in order to keep a RESTful implementation. – Rob Baillie Mar 28 '14 at 9:40
  • 2
    Obviously, query parameters are the best solution, but your question specifically asks about how to deal with filter definitions longer than the limit on URLs imposed by some servers. In order to work around length limit you will either need to compress the query string somehow or you need to use a request method which supports specifying a body of arbitrary length. If you don't want to treat filters as a resource, just support a non-restful interface where filter definitions are POSTed. You will lose cacheability, but if you're data is volatile enough it wouldn't benefit from caching anyway. – pgraham Mar 29 '14 at 14:34
  • You can overcome the need to store filters by simply... not storing them. Nothing about REST guarantees that it is persistent. You might make a request for GET /jobs/37 and receive a result, then someone deletes the resource and 2 seconds later the same request returns a 404. Similarly if you POST /searches and you're redirected to a search result (the search is created and you receive a 201 with Location header to the resource), 2 seconds later that result may be wiped from memory and have to be regenerated. No need for long-term storage. – stevendesu Aug 5 '15 at 17:20
5

This is an old answer but I still can contribute a bit to the discussion. I've observed very often a misunderstanding of REST, RESTful and Architecture. RESTful doesn't ever mention anything about NOT building search, there's nothing in the RESTful about architecture, it's a set of design principles or criteria.

To describe a Search in a better way we have to talk about an architecture in particular and the one that fits better is Resource Oriented Architecture (ROA).

In RESTful there are principles to design, idempotent doesn't mean the result cannot change as I read in some answers, it means the result of an independent request doesn't depend on how many times is executed. It can change, let's imagine I'm continuously updating a database feeding it with some data that is served by a RESTful Api, executing the same GET might change the result but it doesn't depend on how many times it's been executed. If I'm able to freeze the world it means there's no state, transformation, anything inside of the service when I request the resource that leads to a different result.

By definition, a resource is anything that's important to be referenced as a thing by itself.

In a resource oriented architecture (let's call it ROA from now on for brevity) we focus on the resource that could be a lot of things:

  • A version of a document
  • The last updated version of the document
  • A result coming from a search
  • A list of objects
  • The first article I bought from an e-commerce

What it makes it unique in terms of resource is the addresability which means it has only one URI

In that way the search fits perfectly in RESTful considering ROA. We have to use GET because I'm assuming your search is a normal search and it doesn't change anything, so it's idempotent (even if it returns different things depending on new elements added). There's a confusion here in that way because I could stick to RESTful and not to ROA, it means I could follow a pattern that creates a search and return different things with the same parameters because I'm not using the addressability principle of ROA. How is that? Well, if you send the search filters in the body or header the resource is not ADDRESSABLE.

You can find the principles of what's exactly and URI in the W3 original document:

https://www.w3.org/DesignIssues/Axioms

Any URL in this architecture has to be self-descriptive. It's necessary if you follow the principles to address everything in the URI, it means you can use / (slash) to separate whatever you need or query parameters. We know there are limitations on that but this is the architecture pattern.

Following the ROA pattern in RESTful a search is not more than any other resource, the only difference is the resources come from a computation instead of a direct relation to the object itself. Based on the principle I could address and obtain a simple arithmetic calculation service based on the following pattern:

http://myapi.com/sum/1/2

Where sum, 1 and 2 can be modified but the result of the computation is unique and it's adressable, every time I call with the same parameters I obtain the same and nothing changes in the service. The resouce /sum/1/2 and /substract/5/4 stick to the principles perfectly.

3

GET is ok, if you have a static collection that always returns the same results (representation) for one URI. That also implies that the data generating these representations is never altered. The source is a read-only database.

Having GET return different results for one and the same URI violates idempotency/safety and the CoolURI principle and is consequently not RESTful. It is possible to have idempotent verbs write to a database, but they must never affect the representation.

A common search starts with a POST request that returns a reference to the result. It generates the result (it is new and can be fetched with a subsequent GET). This result can be hierarchical (further references with URIs that you can GET), of course, and might reuse elements of earlier searches, if it makes sense for the application.

By the way, I know people do it differently. You don't need to explain to me how convenient it can be to violate REST.

  • Aaaaaaaah - so THAT's how it's supposed to work! Thanks! – Rob Baillie Sep 21 '16 at 11:27
  • 1
    Idempotency doesn't mean it has to always return exactly the same, it has to return the same if NOTHING changes. Search can be considered the result of a computation and it's a resource itself. – Maximiliano Rios Mar 8 '17 at 4:35
  • Idempotency actually DOES mean that the result stays the same. You can, and it is practicable, to use cache control. And you can of course use DELETE that interferes with later GETs. If however, an agent needs to keep knowledge about the application inner workings, it's not RESTful anymore. Above, I was talking about the most extreme idea of REST. In practice, people can violate many aspects of it. They are paying the price when caches don't cache efficiently anymore. – Martin Sugioarto Mar 8 '17 at 7:02
  • "Idempotency actually DOES mean that the result stays the same."... after the request. the point, I believe, is that request doesn't change the data. – AndreiMotinga Aug 13 at 14:27

protected by gnat Mar 8 '17 at 6:09

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.