If URLs are broken, and how to fix them

Question

I have been having trouble clearly defining/specifying how a URL/URI should be applied to situations such as search queries or different resource formats. The question is what an actual "uniform resource identifier" is, and how it should be applied to complex situations involving queries and formats.

The REST paradigm (at least from my history with Ruby on Rails) made it so that you can define clean URLs that got rid of the older URL query parameter style in earlier PHP and related apps. So instead of:

/?format=html&articleId=123

It became:

/articles/123.html

This made it so (what seems like) all URLs could be expressed "RESTfully" without using the ?. So it would be like:

/:class1/:id/:class2/:id/...

You can also get "collections" of resources like this:

/articles

That would return all the articles.

But here's where the idea of URL and REST breaks down. Typically when you return data to /articles, it might be paginated, and might be the most recent articles.

Let's say this is Twitter and there is a constant stream of new content. Then /tweets will constantly return something different than what was returned before. So the question is, what the actual "uniform resource" is.

It seems clear to imagine a "uniform resource" when you are talking about an individual database record, like a user record. You have a list of properties, and an ID:

id: 123
email: [email protected]
name: Example name
...

That would be accessed via /users/123.

However, even this sort of breaks down, when you modify the user record. Say you change the email of the user:

id: 123
email: [email protected]
name: Example name
...

Now the same URL returns something different than before.

To solve this, you start to bring in the idea of versions. So it might be like this:

/users/123/v1
/users/123/v2
...
/users/123/v<hash>

But you don't really see that out in real-world apps. However, that data might be returned in an ETag or other HTTP header, so it is there somewhere, sometimes, but it's not in the URL, the "uniform resource identifier". So the URL is lacking some important uniqueness information.

Another example is with queries. Searching in Google Chome for "StackOverflow" uses this URL:

https://www.google.com/search?q=stackoverflow&oq=stackoverflow&aqs=chrome..123123.abcabc&sourceid=chrome&ie=UTF-8

There is no date or version information there, and the results are tailored to me, so the same URL will return something different for another user.

Another example is with API calls for collections. A facebook URL that provides a cursor to paginated data is closer to a good "URL":

https://graph.facebook.com/me/albums?limit=25&after=MA123123...=

That unique after token probably goes to the database, which stored a set of IDs to use as the next query result if the API call is made, so in a sense it accurately captures the "uniqueness" in URL (unique resource identifier :p). But I'm not sure exactly how Facebook implements that, so it could still be off and not totally accurate in uniquely representing the resource (for example, maybe if new data is added in between now and the usage of that URL, it will return those new records in the result).

The definition of URI from Wikipedia says:

A Uniform Resource Identifier (URI) is a string of characters designed for unambiguous identification of resources and extensibility via the URI scheme.

However, all of the cases presented here (twitter feed, google search results, facebook api response) are ambiguous. Even the simple REST api call to /users/123 is ambiguous because of the versioning changes to the user record.

So my question is, if there is a solution to the problem of using URLs, or something like URLs, to accurately represent a unique resource. That seems like what IPFS and other blockchain technologies are partly doing, creating a unique hash from the content every time it changes (sort of like git). However, this only seems like it would work for objects, not for collections of objects, or api/search results. For that I haven't seen a solution. It is almost as if you want to accurately snapshot the state of the system at a particular time, and use that as a URL. So a search URL would look like this:

/articles?titleMatches=foo&timestamp=123123

And that would return a result set like Facebook's API does, paginated. But then the after cursor would essentially take a snapshot of the database state of all the objects at the current version they are at when that URL request was made. So the after cursor would contain data along the lines of:

articleIdVersionHashes: [<hash1>, <hash2>, ...]

That would make the URL uniquely identify the content.

However, this would break down the meaning of the URL address bar in browsers. Typically you just want to go to https://twitter.com and see your home profile (content under the / URL that is unique to you, or if you are logged out, the Twitter homepage). The URL bar is more of a guideline than a Unique Resource Identifier.

In all of these ways, it seems that the idea of URL is broken. Wondering if there are any solutions to this problem, or alternatives to the idea of a URL, and how to effectively think about the idea of uniquely representing a "resource" (and what a resource actually should mean).

Basically I'm wondering if there is a better way to conceptualize the URL address bar, what it's meaning and purpose is. Because the name "URL" isn't accurately describing what's happening.

Answer 1

I'm afraid that you misunderstood the meaning of “unambiguous identification.” What it means is that an URI points to a specific resource, not that it points to an immutable entity. This also means that a resource may not exist at a given moment (HTTP 404) or may not be accessible in a specific context (HTTP 401), etc.

An URI to your profile identifies precisely and unambiguously the resource which is your profile. It doesn't mean that all the calls to this URI would always result in the exact same response. Performing successive calls over time, you should expect some changes to the profile (for instance after you change your email address).

The unambiguous part, however, ensures that whenever you call this URI, it will point to your profile. If URIs were ambiguous, a call to a same ambiguous URI could give the first time your profile, the second time your bookmarks, the third time the date of your last connection and the fourth time the number of users who used the app today. That is the ambiguity Wikipedia was talking about.

Now to answer your question about how you can represent a given state of a mutable entity, you already have a great solution in your question. Store the snapshots of an entity, and include either the version or the hash of the snapshot in the URI.

Of course, this doesn't mean the exact same call will still always give you the exact same result. For instance, if you did a call yesterday to retrieve a specific state of an entity, doing the same call today can result in, for instance, HTTP 401 if your API key expired during the night, or HTTP 404 if all traces of the entity were removed for legal reasons. If you absolutely need the response to always be the same, HTTP is not a good choice in your case.

Answer 2

There's no such thing as a "RESTful URL". Doesn't exist. REST says nothing about what URLs should look like, and it's designed to treat them as opaque.

Resources in well-designed web APIs often do not map 1-1 to database tables. A web API is an abstraction layer on top of a data store.

You're misunderstanding the acronym. It's not an Identifier of Uniform Resources. It's a Uniform way to Identify Resources. A URI does not specify a resource which never changes. It points to a specific resource, which may have different data over time or for different requesters.

In your example, the resource that /tweets represents is "the N most recent tweets visible to the current user". Requests will always return that conceptual collection of entities. Yes, the tweets will vary over time and by user, but the resource is still what it is.

The resource represented by http://twitter.com is "the current Twitter homepage". That URL uniquely identifies, and always takes you to, the resource that is the current Twitter homepage.

Perhaps it would be helpful to think of a URL as a window in a house. It's always in the same place. It always shows the same part of the outside world. Sometimes there's snow on the ground. Sometimes you see rain, or darkness. But windows don't move, or start showing the front yard instead of the back.

You can lock in the view in a window by taking a picture. That would be adding a timestamp or resource version information into your URL.

Answer 3

The REST paradigm (at least from my history with Ruby on Rails)

That is the wrong starting point - if you want to understand REST, you really need to be starting from Dr Fielding's thesis, specifically his definitions of resources and resource identifiers

a resource R is a temporally varying membership function MR(t), which for time t maps to a set of entities, or values, which are equivalent

Some resources are static in the sense that, when examined at any time after their creation, they always correspond to the same value set. Others have a high degree of variance in their value over time.

The only thing that is required to be static for a resource is the semantics of the mapping, since the semantics is what distinguishes one resource from another.

Note that PUT /target-resource and PATCH /target-resource have semantics that should make it clear that resource identifiers do not necessarily map to representations that are fixed for all time.

Now, from the perspective of "REST", these two URI are not necessarily related

/users/123
/users/123/v1
/users/123/sha=5bb6ed2db6b5b9fa91f82dc6e892f2631aa84b76

To a generic client, these identify different resources, in particular, they each have their own caching meta data. We can include in the meta data caching headers that tell the client that the representation returned by the first identifier is valid only for a short period of time, but that the later resources are valid for much longer periods.

This is done -- not by the spelling of the identifier, but by the headers provided with the representation by the origin server.

There are even facilities available for informing the client that the requested resource currently has the same representation as another. You'll want to review the details of the Content-Location header yourself, but the basic idea is there.

However, do notice that the capabilities for doing this are encoded into the metadata, not the identifier. There's a lot more to a message than just the target uri.

Sidebar:

However, this only seems like it would work for objects, not for collections of objects, or api/search results.

No, it will work just fine for collections. Review wikipedia's introduction to the Merkel Tree, or a good tutorial on the git object model.