6

I'm building an API endpoint for a UI grid to search, filter, and display a list of domain objects, let's call them "widgets." In the past, I would have built this with a list of named query string parameters, like this:

GET /api/v1/widgets?type=2&name=what&from=2019-12-31&to=2020-01-03&pagesize=25&page=2&sort=name,-createdate

This would result in SQL something like

SELECT <selectlist>
FROM widget
WHERE type = 2
    AND name LIKE 'what%'
    AND createdate >= '2019-12-31'
    AND createdate <= '2020-01-03'
ORDER BY name ASC, createdate DESC
LIMIT 25, 25;

I've had a co-worker propose that instead of a long list of parameters, we pass a couple of JSON objects on the query string, like this:

"filters": {
    { "column": "type", "operator": "=", "value": 2 },
    { "column": "name", "operator": "like", "value": "what" },
    { "column": "createdate", "operator": ">=", "value": "2019-12-31" },
    { "column": "createdate", "operator": "<=", "value": "2020-01-03" }
},
"pagination": {
    "page": "2",
    "pagesize": "25",
    "sort": [ "name", "createdate" ],
    "sortdirection": [ "asc", "desc" ]
}

Which, after encoding, looks something like this as a URL (sorry if I messed up the encoding, I made this up as an example):

GET /api/v1/widgets?filters=[%7B%22column%22:%22type%22,%22operator%22:%22%3D%22,%22value%22:2%7D,%7B%22column%22:%22name%22,%22operator%22:%22like%22,%22value%22:%22what%22%7D,%7B%22column%22:%22createdate%22,%22operator%22:%22%3E%3D%22,%22value%22:%222019-12-31%22%7D,%7B%22column%22:%22createdate%22,%22operator%22:%22%3C%3D%22,%22value%22:%222020-01-03%22%7D]&pagination=%7B%22page%22:%222%22,%22pagesize%22:%2225%22,%22sort%22:[%22name%22,%22createdate%22],%22sortdirection%22:[%22asc%22,%22desc%22]%7D

We are in JavaScript on both client and server, so parsing the JSON object is not a difficult task. And I realize that the structure of the JSON object for querying would have to be altered, as it does not account for some things. Disregarding that, the basic question is:

Is this a bad/good idea? I see advantages and disadvantages, but I'm trying to look past my personal bias.

  • What advantages / disadvantages can you see? – richzilla Jan 3 at 15:37
  • @richzilla - I'm biased against it. As you pointed out, query string length is an issue, and I also really dislike how it makes the query string "inaccessible." I like API URLs that are clearly readable. I also think that this approach requires too much knowledge of the inner workings of the API by the client, which introduces tight-coupling. The only advantage I really see is that it makes it easier to code, though I think the code ends up being more abstract and less maintainable. – AJ Johnson Jan 3 at 15:42
  • 1
    Primarily I'm wondering if I'm being old fashioned ;-) – AJ Johnson Jan 3 at 15:44
  • This looks like a very verbose and unreadable format for a query string. Looking at the structure, I'd perhaps suggest Django-like format would be the most concise and very readable: example.com/?type=2&name__like=what&createdate__ge=2019-12-31&createdate__le=2020-01-03&pagesize=25&page=2&sort=name,-createdate or if you want to avoid writing custom parser, use csv parser to parse the value that looks like example.com/?filter[]=eq:type:2&filter[]=like:name:what&filter[]=ge:createdate:2019-12-31& filter[]=le:createdate:2020-01-03&pagesize=25&page=2&sort=name,-createdate. – Lie Ryan Jan 6 at 11:05
  • Or if you want to go full blown silly and have a full fledged frontend querying language, there's also the GraphQL path, which has the benefit of actually being well specified and have third party implementations. – Lie Ryan Jan 6 at 11:11
15

Its not a brilliant idea. The URI is not really a good place for data of unpredictable length, and although there is no 'official' maximum length, many webservers apply their own limit (IIS is 2083 characters, for example). Some webservers also have restrictions on acceptable characters in URIs.

There are also other considerations that are generic to URIs. Are the contents of the query in any way sensitive? Many servers will log URIs and query string parameters which may leave sensitive data in places you dont want it.

For this kind of variable length data, using HTTP Post would be best solution.

If you did still want to use the query string, another option would be to base64 the JSON string, meaning you wouldnt need to worry about escaping the spaces and special characters. This would mean the URI was not 'human readable', but I would argue that URI encoded JSON isn't particularly easy to read either.

  • Excellent point about query contents being sensitive. Thank you! – AJ Johnson Jan 3 at 15:45
  • 4
    Base64 isn't secure anyway. It's merely obfuscation. – Robert Harvey Jan 3 at 20:31
  • 2
    Note that you may want to use modified "url-safe Base64" to avoid unnecessary %-ending as regular character set for Base64 includes + and = which have special meaning for query params. – Alexei Levenkov Jan 4 at 1:26
4

What's bad is passing massive amounts of data in the URI, so a POST would be better in my opinion. Because of possible limitiations in size, because of problems passing arbitrary data, because of security concerns. That's of course a simple change. You generate the JSON, then you decide how to send it.

Check with the database guys that it is not difficult to generate efficient queries from the data in your JSON. I assume it isn't. And generating the JSON shouldn't be difficult. JSON gives you a chance to create more complex queries if you feel the need.

  • Thanks. I'm the "database guy" as well (small company ;-) ). This endpoint will have a "create" operation, too, so I was trying to avoid a POST as that, traditionally, is used for creating new objects. I do agree that using JSON gives us more flexibility, though I worry that the UI needs to know too much about the query structure. Perhaps that's my old-fashioned bias showing, though. – AJ Johnson Jan 3 at 17:24
  • 1
    @AJJohnson Semantically you can argue either way, that a search is just "getting" data that already exists, but you can also argue that a "Search" resource exists, and POSTing to a search endpoint "creates" a search resource. The latter option makes more sense if searching is expensive and you want to store the results of the search after they are gathered so as not to have to recompute the same search again. – Jeff Lambert Jan 3 at 20:10
  • 2
    Database should not be relevant to this. JSON or not, you're passing parameters to some backend API. Whether they come in as JSON or not shouldn't have much to do with the database query, unless you're actually passing the JSON string itself into a query... that would be... inadvisable. – Brandon Jan 4 at 2:32
  • 3
    @AJJohnson Knowing that data exists is different from knowing how it's stored. An API to search temperatures might take either Celsius or Fahrenheit as parameters, while storing only in Kelvin. The UI and the backend MUST agree on an interface. Each one needs to know nothing more about what happens on the other side. – CJ Dennis Jan 4 at 2:53
1

I went through almost the same analysis with a former coworker. He was adding a new API endpoint and wanted to pass in a complex JSON object into the query string instead of separate parameters.

Like yourself, I had reservations.

Exactly what the pros and cons are will depend on your specific product needs, architecture, team structure, and other factors.

In our case, I found the tradeoffs were simply:

Pro: Saved half a line of code in the API implementation

Cons:

  • Swagger (our API documentation tool) would have no intuitive way to document the structure of this API. API documentation, accessibility, and discoverability were important to us.
  • Inconsistent with the rest of the world. Adhering to common conventions and expectations simplifies things and helps avoid mistakes.

That said, I don't think this pattern is necessarily bad. In certain cases, like building up a dynamic set of search filters, it might make sense. Your use case might be a good candidate for this.

I don't totally agree with the pushback against GET that you have received in other answers. GET is the semantically correct HTTP verb because you are getting data. You're not mutating any state. It's idempotent and cacheable. POST is not.

GET requests can accept a body, just like POST requests and POST requests can accept a query string. There is a misconception that GET means query string and POST means body. They are orthogonal.

Factors I'd consider in your situation:

  • Documentability, if this is important to your organization and/or product
  • Tooling - Most HTTP routing frameworks have middleware to automatically parse query string parameters and expose them as a map. For example, in a node.js express app, I can check req.query.filters to get the filters. You'd have to do your own parsing if you go the JSON route, but that's not hard. It's a simple JSON.parse call.
  • Validation - There are modules which can automatically validate a request based on a schema you supply. Moving your inputs to a JSON object may create a black box, forcing you to validate inputs yourself (and we all know that inadequate input validation is one of the leading causes of security vulnerabilities)
  • Complexity - Do you really need to support infinite filters or even a dynamic list of filters? Do the operators need to be configurable like that? This JSON object design is highly extensible. If there are only 4 things to filter on and they always work the same way, hard coding will save you a ton of time.
  • Testing overhead - following from the complexity point above, every combination of criteria needs to be tested. If your API allows for any operator across any field, you need to test each one of these cases. If your clients only use the API a single way, then you have a bunch of unused use cases you are stuck supporting. For example, if the frontend always performs a wildcard search, then testing the = case for name is a waste because it's never really used, but your API supports it, so it better work.

Despite all my concerns with this approach, Elasticsearch does the JSON pattern and it works quite well for them. But Elasticsearch is a search engine. The set of filters being passed in, operators, etc. need to be dynamic because the JSON structure is actually meant to be a search query. Elasticsearch supports any sort of user-defined schema, so they expose a general query language as JSON.

If you have a few inputs on a web page and they map directly to known SQL predicates, you may be going overkill with the JSON pattern.

This is why it really matters what your product actually is.

Solve for the problem you have, not a problem that you may have some day.

  • "GET requests can accept a body". It's more complicated than that. The various HTTP RFCs for years contained language like "the message-body SHOULD be ignored when handling the request". RFC7231 now says that sending a body has no defined semantics. OpenAPI for that reason does not allow you to document a body with GET. In general many web frameworks will have some roundabout way to get to the body of a GET, but it's not the default and it's likely to break if there are any proxies or similar things between client and server. All in all it's a bad idea. – Voo Jan 4 at 11:11
  • Thank you for your comprehensive, well-written response. This is what I was looking for, though I do agree that GET with a body isn't acceptable for us. But we are using Swagger as well, and I very much agree with you on your points regarding complexity, testing, and validation. And "solve for the problem you have" is very much akin to my favorite programming axiom, YAGNI. – AJ Johnson Jan 4 at 17:19
0

Note that you don't need to (see RFC 7320 section 2.4) use the standard form data format (with & and =) (see the URL Standard section 5) for the query string, you can just make up a format as long as it only uses allowable characters (letters, numbers, -, ., _, ~, !, $, &, ', (, ), *, +, ,, ;, =, :, @, /, and ? (see RFC 3986)).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.