68

Compared to about 10 years ago I have noted a shift towards frameworks using the style of routing that decouples the URL path from the filesystem. This is typically accomplished with the help of a front-controller pattern.

Namely, when before, URL path was mapped directly to the file system and therefore reflected exact files and folders on disk, nowadays, the actual URL paths are programmed to be directed to specific classes via configuration, and as such, no longer reflect the file system folder and file structure.

Question

How and why did this become commonplace? How and why was it decided that it's "better" to the point where once-commonplace direct-to-file approach was effectively abandoned?

Other Answers

There is a similar answer here that goes a bit into the concept of route and some benefits and drawbacks: With PHP frameworks, why is the "route" concept used?

But it does not address historical change aspects, or how or why this change gradually happened, to where any new projects nowadays are pretty much using this new routing style pattern and direct-to-file is outdated or abandoned.

Also, most of those benefits and drawbacks mentioned, do not appear to be significant enough to warrant such a global change. The only benefit that I can see driving this change perhaps is hiding the file/folder system from end-user, and also lack of ?param=value&param2=value, which makes URLs look a tad cleaner. But were those the sole reason for the change? And if yes, why were those reasons behind it?

Examples:

I am most familiar with PHP frameworks and many popular modern frameworks use this decoupled routing approach. To make it work you set up URL rewriting in Apache or similar web server, to where web application functionality is typically no longer triggered via a direct-to-file URL path.

Zend Expressive

https://docs.zendframework.com/zend-expressive/features/router/aura/
https://docs.zendframework.com/zend-expressive/features/router/fast-route/
https://docs.zendframework.com/zend-expressive/features/router/zf2/

Zend Framework

https://docs.zendframework.com/zend-mvc/routing/

Laravel

https://laravel.com/docs/5.5/routing

CakePHP

https://book.cakephp.org/3.0/en/development/routing.html

  • 14
    was there really such a change? or maybe most languages/frameworks never used direct filesystem mapping? maybe it's just PHP that'is catching up with the rest of sane? your observations are in my opinion wrong, so there is no good answer. also "shift" you mention occured only in PHP world. but it's too broad question in my opinion... – rsm Jan 5 '18 at 16:55
  • 2
    @rsm I had a similar first reaction but on further thought, it really is something that has been done across many languages and platforms leading to it being a really common source of vulnerabilities. – JimmyJames Jan 5 '18 at 17:00
  • 6
    @rsm, this may be more evident in PHP frameworks, but to use another way -- at a certain time, before any framework really caught on, be it ASP, .NET, PHP, JSP, etc, the web mostly used direct-to-file approach. Why did all these frameworks develop to use the decoupled approach? Technically, direct-to-file approach is still feasible, and I am sure modern frameworks can use it. Or can they? Maybe they can't, and maybe there are good reasons as to why they don't? What would those reasons be..? They don't even provide a way or a plugin to do this, they just eradicated direct-to-file altogether. – Dennis Jan 5 '18 at 17:10
  • 2
    Isn't this (in part) also about viewing an URL (a locator, how to find a resource) as URI (and identifier)? – Hagen von Eitzen Jan 6 '18 at 16:39
  • 2
    Related reading from ancient history: Cool URIs Don't Change - Tim Berners-Lee, 1998 – Michael Hampton Jan 7 '18 at 2:26

10 Answers 10

72

In its most basic form, a website serves static files. Mapping the URL path to a file path is the most obvious choice; essentially, it's a read-only FTP site.

Then people wanted to change the content of the page with some scripting. The easiest way is to embed a scripting language into the page and run it through an interpreter. Again, given the already existing path -> file path routing, this was simple enough.

But really, you are running that file as an argument to the interpreter now. You have to identify when the request is for a static file and when it's for something that you need to interpret.

Once you start to use more advanced compiled languages, you are even more divorced from the file location.

Plus, your web server is already caching static files and doing all sorts of optimizations, that means hitting the file system is the exception rather than the rule. At this point, the old link file system path is more of a hindrance than a help.

But I think the real sea change came when users wanted to get rid of the file extension from the path. Getting myPage.asp or myPage.php was somthing that confused 'normal' people and interfered with SEO.

Because the user sees the path, it has become part of the UI of the web, and as such, it needs to be completely free of any technical limitations. We have lost the 'www' and virtually everything is a '.com'. Multiple URLs will point to the same page.

If I make more money with mydomain.com/sale vs www.mydomain.co.uk/products/sale.aspx, then I don't want any technical limitations to stand in my way.

  • 7
    I always thought the desire to hide file extensions was in part "security by obscurity" - making it a bit harder to tell what technology was employed by a site in order to make it less easy to target for particular attached/known exploits of certain servers and techs at the time – Caius Jard Jan 5 '18 at 17:39
  • 20
    @CaiusJard that's part of it, but another part is technology agnosticism - having replaced file.html in our paths, we didn't want to get stuck with another switch later (for example, file.phtml to file.php, or even to file.asp). But since we're dissociating URL and filesystem path (using routing or whatever) to access resources built from database records and/or other sources, why have an extension in the URL at all? – HorusKol Jan 5 '18 at 18:03
  • 39
    @HorusKol Technology agnosticism isn't even just about having to change all the files in your path. Being able to change backend technologies without breaking your customer's workflow and bookmarks and without destroying your SEO can be huge. – Shane Jan 5 '18 at 18:51
  • 7
    Interestingly it was never recommended to have file name extensions in the URI, so they should have been decoupled from the file system early on. The earliest reference I can find for this is from 1998, which actually predates most of the developments described in this answer. – Konrad Rudolph Jan 5 '18 at 19:46
  • 7
    In the old days mydomain.com/sale still worked; it redirected to /sale/ and loaded just fine (your page was mydomain.com/sale/index.aspx but nobody ever saw the index.aspx). – Joshua Jan 5 '18 at 20:55
39

You can look to a white paper by Roy Fielding on REpresentational State Transfer (REST) as to the when and the why. The first framework I was aware of that made the distinction between a resource and a file was Ruby on Rails--introducing the concept of URL to code routing.

The main concepts behind REST that were transformational were:

  • A URL represents a resource
  • That resource can have multiple representations
  • The URL should not break if the application is restructured
  • Applications should embrace the statelessness of the web

The main drawback of having files be served directly by the URL is that you experience the following issues:

  • Resource links are constantly broken as web-sites are reorganized
  • Resource and representation are tied together

I think it is important to provide some fair balance as well:

  • Not all resources are equal in importance. That's why you still have style based resources served directly (CSS, JavaScript/EcmaScript, images)
  • There are refinements of REST like HATEOAS that better support single page apps.
  • when you say representation do you mean things like JSON/HTML/TEXT/etc? I'm vaguely familiar with REST, but I imagine even with REST you have to have some kind of a trigger to change response representation... – Dennis Jan 5 '18 at 17:48
  • @Dennis, yes. HTTP has a number of headers that can be used to hint at the desired form (developer.mozilla.org/en-US/docs/Web/HTTP/Content_negotiation) and REST was all about embracing the strengths of HTTP. However, it's still not uncommon for an app to have a proprietary way of negotiating the desired content. – Berin Loritsch Jan 5 '18 at 17:51
  • 5
    CGI (1993), Servlets (1997) and JSP (1999) often decoupled URLs from the filesystem, and pre-date REST (2000). However this answer is basically correct in identifying the reasons for the popularity of the design pattern: REST, Java Struts and Ruby on Rails have a huge influence on the 21st century popularity of separating resources from representations. – dcorking Jan 6 '18 at 17:39
  • 1
    According to Fielding's paper, "The first edition of REST was developed between October 1994 and August 1995" – Connor Jan 7 '18 at 0:20
  • 1
    @dcorking, CGI at that time didn't decouple URLs from files, it just ran the file instead 9 times out of 10. Servlets might be the closest match, but if you are talking about the concept of routes and having a designed url space, that came with Rails and frameworks like it. – Berin Loritsch Jan 7 '18 at 14:21
20

I don't think it's an artefact of modern web application frameworks, it's mostly an artefact of dynamic page serving in general.

In the old days there were mostly static web pages, where a software served individual files from the file system by their path. They did so mostly because the 1:1 mapping of URL paths to file system paths (with one directory designated as the web root) was the obvious choice, though URL rewriting (e.g. to make redirects after moving files) was common as well.

Then came the age of serving dynamic content. CGI scripts (and everything evolved from them) did create the pages on the fly, being backed by a database of some kind. GET parameters in URL became commonplace, for example en.wikipedia.org/w/index.php?title=Path_(computing).

However it is more userfriendly to have a readable URL consisting of only path segments. So the dynamic applications mapped simple paths (for example en.wikipedia.org/wiki/Path_(computing)) to parameters, and these mappings are known as "routes".

Maybe this approach feels more recent as it gained popularity when the importance of usability was recognised more widely, and also became part of SEO. This is probably the reason why it was built directly into the large web frameworks.

12

One reason is that loading a file from disk on every request is slow, so web servers started creating ways to cache files in memory, then if you're going to try to keep it in memory anyway, why does it matter where it was on disk?

One reason is that many web frameworks are written in compiled languages, so you don't even have a file structure on disk, just a jar file or whatever. Interpreted languages borrowed ideas they liked from the compiled ones.

One reason is the desire for more semantic, dynamic routes, like https://softwareengineering.stackexchange.com/questions/363517/how-and-why-did-modern-web-application-frameworks-evolve-to-decouple-url-routes. Obviously, you don't want a /var/www/questions/363517/how-and-why-did-modern-web-application-frameworks-evolve-to-decouple-url-routes.php file. You used to make url-rewriting rules in the web server configuration to create routes like this. Now it's just a code change, which is much simpler operationally.

  • You don't need url rewriting for that example. – Yay295 Jan 5 '18 at 22:07
  • 1
    It's handled by code that recognises the first part of the path, and uses the number /363517/ to look up the question in a database. Nothing to do with the web server itself, but an application ... – Will Crawford Jan 7 '18 at 23:59
11

On of the major reasons is likely that this approach of mapping URIs to file paths has lead to a large number of accidental releases of data via File Path Traversal

When you map the path to the file system, it means that you then need to check that every path that you receive as a request maps to files that should be accessible through to clients. A simple approach to guarantee that is not happening is to eliminate the transparent mapping and do it more explicitly.

This is not a PHP-only issue. As evidence here is a relevant section of an Apache hardening guide.

  • 1
    Why the downvote? – JimmyJames Jan 5 '18 at 17:19
8

I can't answer for the industry, but I can tell you why I moved away from the URL = file system back in the early 2000's towards virtual 'routes'.

Working with 'old school' PHP, if you have a 1000 PHP pages, you'd have 1000 PHP files representing those pages. Each one duplicating header/footer includes and possibly some other logic. Now let's say you need to change that. What a mess you now have on your hands! You either have to change all 1000 files, or you end up with a jumble of very ugly code in the header/footers to handle all the cases. Using virtual routes, your header/footer logic, database connection logic, and other initialization are included once, period. Much better to work with.

Another reason is to avoid ambiguity. As applications grow, the headers/footers that get included become more complex. They usually had nested includes of their own that depended on various things. In the PHP file for the 'page', often you encountered ambiguity about whether a variable isset() or not. Using virtual routes, and an application where everything you need is loaded on each page load, you no longer have that concern.

Lastly (though there are other reasons, but it's the last I'll list), many of those 1000 pages represent code that would be duplicated. So after refactoring into a proper set of classes and templates, the code is greatly simplified and you can do everything you want to do without having those 1000 files.

  • can you say more why you would end up with very ugly code? I can see the need to change 1000 files (assuming it is to update header/footer includes), but what do you mean by jumble of ugly code? – Dennis Jan 5 '18 at 21:33
  • See the paragraph I just added. But basically, as you extend the header/footer/initialization code to handle more cases, especially if you conditionally include other files (it was a bad habit, but lots of PHP programmers did it), you end up with very difficult to follow code. – GrandmasterB Jan 5 '18 at 21:37
5

I won't go into too much detail as to why this separation is beneficial. The main argument is that it separates semantics (what are you actually trying to access) from the underlying implementation.

Taking that the benefits outweigh the costs as a given - which would be a separate question - it's not difficult to see why it was gradually adopted. I don't think there's a single event that caused this, although I'd certainly be open to being educated on this.

At least in my experience, initially this was often done via the Apache configuration - and presumably other web servers supported this as well. However, conceptually there is no good reason why the server should be tasked with this. After all, the routes are specific to the actual application, so it makes sense to define them there.

This did change globally, but as you point out, gradually. The reason for this is almost certainly a very simple one: good ideas spread over time. This is also why it's not such a surprise that the change happened globally. It's not that everybody got together and decided to do it this way. Rather, every project adapted this approach when they thought it would be beneficial (and projects that didn't support it eventually disappeared).

1

The RFCs already built the concepts from the ground up, with URIs (that didn't attach any semantics to the local part), and URLs as a special case that introduced path-like semantics to allow HTML documents to use links relative to the document base URL.

The obvious implementation is to map the local part of the URL directly to the file system, so this is what simple setups did — whether you use a dedicated relational database to look up a document, or take advantage of the highly optimized low-overhead key-value store you already have does not matter to the outside, but certainly affects your cost structure for serving the documents.

If you have a web application with persistent data, that cost structure changes: you always have the overhead of running the application, and integrating URL decoding into it makes a lot of features easier to implement, reducing cost.

1

At the beginning of time, URLs mapped directly to file paths on the server because it's easy, and there's no other way of doing it anyway, is there? If I ask for /path/to/index.php, I'll get /path/to/index.php starting from the root directory of the website (usually not of the server itself, the website should be kept in a directory or a subdirectory further down).

Then after a couple of years, we started learning about rewriting, that is serving a different resource than the one that was apparently asked for. /request/path/to/index.php can actually serve /response/path/to/index.php.

Another trick is hiding index.php. If I ask for /index.php?foo=bar&baz=qux the server can respond by hiding index.php like so: /?foo=bar&baz=qux, all the while actually serving index.php anyway.

The next step, which is the big important one, is that we learnt to redirect all URLs to /index.php. So now, /path/to/some/page is silently redirected to /index.php?path/to/some/page. This is a bit tricky, because normally each slash represents a new subdirectory, but in this case the web server is configured to send the path as a parameter, instead of looking in it.

Now that we have this, we need a completely different way of thinking about how the website is organised. Before, it was a loose collection of different pages. Now, everything is routed through a single entry page. This makes the site much more complicated, but provides opportunities that weren't available before, such as site-wide user authentication, uniform application of headers, footers and styles, etc.

It effectively turns your hundred or thousand app website (if you consider each file to be its own app) into a single, much more complicated but much more consistent app.

This is a huge leap, as you can no longer tell what code will be executed simply by looking at the URL. You now need to have a deep understanding of how your particular framework translates URL paths into code paths, and although there are similarities between frameworks, most are different enough that you need some familiarity to be able to work with the code.

Long story short, it was a gradual evolution of discovery, not a sudden jump, and each developer pretty much had to go through the same journey of discovery. The learning curve is pretty steep, unless you can grasp abstract concepts really quickly.

-1

As a long-time webdev, I think the advent of navigation-less history control (history.pushState()) around the time of HTML5 made this practical. Before that, you had to reload the page to update the URL bar, unless you only updated the fragment (/path#fragment). This fragment was invisible to the server (it's not routed), so the only way to refresh or bookmark a dynamic page was via JavaScript.

This has major implications for SEO, and led google to develop a seldom-used "hashbang" schema that required a server-side mapping of dynamic hashes to physical URLs. This was unwieldy and not universal among robots, leading the (false) axiom: "spiders can't crawl ajax content". But the benefits of ajax content are tangible: try using google maps w/o JS for example.

The solution was a way to update the URL bar with a value that can be mirrored on the server (allowing bookmarks and JS-less refreshing), WITHOUT reloading the page. Once this capability was available, developers could "navigate" a site by simply updating a "main content section", URL bar, and breadcrumbs. This meant that all the JS+CSS didn't need refetched+parsed, allowing a MUCH faster page-to-page transfer.

protected by gnat Jan 7 '18 at 22:02

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.