How and why did modern web application frameworks evolve to decouple URL routes from the file system?

Question

Compared to about 10 years ago I have noted a shift towards frameworks using the style of routing that decouples the URL path from the filesystem. This is typically accomplished with the help of a front-controller pattern.

Namely, when before, URL path was mapped directly to the file system and therefore reflected exact files and folders on disk, nowadays, the actual URL paths are programmed to be directed to specific classes via configuration, and as such, no longer reflect the file system folder and file structure.

Question

How and why did this become commonplace? How and why was it decided that it's "better" to the point where once-commonplace direct-to-file approach was effectively abandoned?

Other Answers

There is a similar answer here that goes a bit into the concept of route and some benefits and drawbacks: With PHP frameworks, why is the "route" concept used?

But it does not address historical change aspects, or how or why this change gradually happened, to where any new projects nowadays are pretty much using this new routing style pattern and direct-to-file is outdated or abandoned.

Also, most of those benefits and drawbacks mentioned, do not appear to be significant enough to warrant such a global change. The only benefit that I can see driving this change perhaps is hiding the file/folder system from end-user, and also lack of ?param=value&param2=value, which makes URLs look a tad cleaner. But were those the sole reason for the change? And if yes, why were those reasons behind it?

Examples:

I am most familiar with PHP frameworks and many popular modern frameworks use this decoupled routing approach. To make it work you set up URL rewriting in Apache or similar web server, to where web application functionality is typically no longer triggered via a direct-to-file URL path.

Zend Expressive

https://docs.zendframework.com/zend-expressive/features/router/aura/
https://docs.zendframework.com/zend-expressive/features/router/fast-route/
https://docs.zendframework.com/zend-expressive/features/router/zf2/

Zend Framework

https://docs.zendframework.com/zend-mvc/routing/

Laravel

https://laravel.com/docs/5.5/routing

CakePHP

https://book.cakephp.org/3.0/en/development/routing.html

Answer 1

In its most basic form, a website serves static files. Mapping the URL path to a file path is the most obvious choice; essentially, it's a read-only FTP site.

Then people wanted to change the content of the page with some scripting. The easiest way is to embed a scripting language into the page and run it through an interpreter. Again, given the already existing path -> file path routing, this was simple enough.

But really, you are running that file as an argument to the interpreter now. You have to identify when the request is for a static file and when it's for something that you need to interpret.

Once you start to use more advanced compiled languages, you are even more divorced from the file location.

Plus, your web server is already caching static files and doing all sorts of optimizations, that means hitting the file system is the exception rather than the rule. At this point, the old link file system path is more of a hindrance than a help.

But I think the real sea change came when users wanted to get rid of the file extension from the path. Getting myPage.asp or myPage.php was somthing that confused 'normal' people and interfered with SEO.

Because the user sees the path, it has become part of the UI of the web, and as such, it needs to be completely free of any technical limitations. We have lost the 'www' and virtually everything is a '.com'. Multiple URLs will point to the same page.

If I make more money with mydomain.com/sale vs www.mydomain.co.uk/products/sale.aspx, then I don't want any technical limitations to stand in my way.

Answer 2

You can look to a white paper by Roy Fielding on REpresentational State Transfer (REST) as to the when and the why. The first framework I was aware of that made the distinction between a resource and a file was Ruby on Rails--introducing the concept of URL to code routing.

The main concepts behind REST that were transformational were:

• A URL represents a resource
• That resource can have multiple representations
• The URL should not break if the application is restructured
• Applications should embrace the statelessness of the web

The main drawback of having files be served directly by the URL is that you experience the following issues:

• Resource links are constantly broken as web-sites are reorganized
• Resource and representation are tied together

I think it is important to provide some fair balance as well:

• Not all resources are equal in importance. That's why you still have style based resources served directly (CSS, JavaScript/EcmaScript, images)
• There are refinements of REST like HATEOAS that better support single page apps.

Answer 3

I don't think it's an artefact of modern web application frameworks, it's mostly an artefact of dynamic page serving in general.

In the old days there were mostly static web pages, where a software served individual files from the file system by their path. They did so mostly because the 1:1 mapping of URL paths to file system paths (with one directory designated as the web root) was the obvious choice, though URL rewriting (e.g. to make redirects after moving files) was common as well.

Then came the age of serving dynamic content. CGI scripts (and everything evolved from them) did create the pages on the fly, being backed by a database of some kind. GET parameters in URL became commonplace, for example en.wikipedia.org/w/index.php?title=Path_(computing).

However it is more userfriendly to have a readable URL consisting of only path segments. So the dynamic applications mapped simple paths (for example en.wikipedia.org/wiki/Path_(computing)) to parameters, and these mappings are known as "routes".

Maybe this approach feels more recent as it gained popularity when the importance of usability was recognised more widely, and also became part of SEO. This is probably the reason why it was built directly into the large web frameworks.

Answer 4

One reason is that loading a file from disk on every request is slow, so web servers started creating ways to cache files in memory, then if you're going to try to keep it in memory anyway, why does it matter where it was on disk?

One reason is that many web frameworks are written in compiled languages, so you don't even have a file structure on disk, just a jar file or whatever. Interpreted languages borrowed ideas they liked from the compiled ones.

One reason is the desire for more semantic, dynamic routes, like https://softwareengineering.stackexchange.com/questions/363517/how-and-why-did-modern-web-application-frameworks-evolve-to-decouple-url-routes. Obviously, you don't want a /var/www/questions/363517/how-and-why-did-modern-web-application-frameworks-evolve-to-decouple-url-routes.php file. You used to make url-rewriting rules in the web server configuration to create routes like this. Now it's just a code change, which is much simpler operationally.

Answer 5

On of the major reasons is likely that this approach of mapping URIs to file paths has lead to a large number of accidental releases of data via File Path Traversal

When you map the path to the file system, it means that you then need to check that every path that you receive as a request maps to files that should be accessible through to clients. A simple approach to guarantee that is not happening is to eliminate the transparent mapping and do it more explicitly.

This is not a PHP-only issue. As evidence here is a relevant section of an Apache hardening guide.

Answer 6

I can't answer for the industry, but I can tell you why I moved away from the URL = file system back in the early 2000's towards virtual 'routes'.

Working with 'old school' PHP, if you have a 1000 PHP pages, you'd have 1000 PHP files representing those pages. Each one duplicating header/footer includes and possibly some other logic. Now let's say you need to change that. What a mess you now have on your hands! You either have to change all 1000 files, or you end up with a jumble of very ugly code in the header/footers to handle all the cases. Using virtual routes, your header/footer logic, database connection logic, and other initialization are included once, period. Much better to work with.

Another reason is to avoid ambiguity. As applications grow, the headers/footers that get included become more complex. They usually had nested includes of their own that depended on various things. In the PHP file for the 'page', often you encountered ambiguity about whether a variable isset() or not. Using virtual routes, and an application where everything you need is loaded on each page load, you no longer have that concern.

Lastly (though there are other reasons, but it's the last I'll list), many of those 1000 pages represent code that would be duplicated. So after refactoring into a proper set of classes and templates, the code is greatly simplified and you can do everything you want to do without having those 1000 files.

Answer 7

I won't go into too much detail as to why this separation is beneficial. The main argument is that it separates semantics (what are you actually trying to access) from the underlying implementation.

Taking that the benefits outweigh the costs as a given - which would be a separate question - it's not difficult to see why it was gradually adopted. I don't think there's a single event that caused this, although I'd certainly be open to being educated on this.

At least in my experience, initially this was often done via the Apache configuration - and presumably other web servers supported this as well. However, conceptually there is no good reason why the server should be tasked with this. After all, the routes are specific to the actual application, so it makes sense to define them there.

This did change globally, but as you point out, gradually. The reason for this is almost certainly a very simple one: good ideas spread over time. This is also why it's not such a surprise that the change happened globally. It's not that everybody got together and decided to do it this way. Rather, every project adapted this approach when they thought it would be beneficial (and projects that didn't support it eventually disappeared).

Answer 8

The RFCs already built the concepts from the ground up, with URIs (that didn't attach any semantics to the local part), and URLs as a special case that introduced path-like semantics to allow HTML documents to use links relative to the document base URL.

The obvious implementation is to map the local part of the URL directly to the file system, so this is what simple setups did — whether you use a dedicated relational database to look up a document, or take advantage of the highly optimized low-overhead key-value store you already have does not matter to the outside, but certainly affects your cost structure for serving the documents.

If you have a web application with persistent data, that cost structure changes: you always have the overhead of running the application, and integrating URL decoding into it makes a lot of features easier to implement, reducing cost.

Answer 9

At the beginning of time, URLs mapped directly to file paths on the server because it's easy, and there's no other way of doing it anyway, is there? If I ask for /path/to/index.php, I'll get /path/to/index.php starting from the root directory of the website (usually not of the server itself, the website should be kept in a directory or a subdirectory further down).

Then after a couple of years, we started learning about rewriting, that is serving a different resource than the one that was apparently asked for. /request/path/to/index.php can actually serve /response/path/to/index.php.

Another trick is hiding index.php. If I ask for /index.php?foo=bar&baz=qux the server can respond by hiding index.php like so: /?foo=bar&baz=qux, all the while actually serving index.php anyway.

The next step, which is the big important one, is that we learnt to redirect all URLs to /index.php. So now, /path/to/some/page is silently redirected to /index.php?path/to/some/page. This is a bit tricky, because normally each slash represents a new subdirectory, but in this case the web server is configured to send the path as a parameter, instead of looking in it.

Now that we have this, we need a completely different way of thinking about how the website is organised. Before, it was a loose collection of different pages. Now, everything is routed through a single entry page. This makes the site much more complicated, but provides opportunities that weren't available before, such as site-wide user authentication, uniform application of headers, footers and styles, etc.

It effectively turns your hundred or thousand app website (if you consider each file to be its own app) into a single, much more complicated but much more consistent app.

This is a huge leap, as you can no longer tell what code will be executed simply by looking at the URL. You now need to have a deep understanding of how your particular framework translates URL paths into code paths, and although there are similarities between frameworks, most are different enough that you need some familiarity to be able to work with the code.

Long story short, it was a gradual evolution of discovery, not a sudden jump, and each developer pretty much had to go through the same journey of discovery. The learning curve is pretty steep, unless you can grasp abstract concepts really quickly.

Answer 10

As a long-time webdev, I think the advent of navigation-less history control (history.pushState()) around the time of HTML5 made this practical. Before that, you had to reload the page to update the URL bar, unless you only updated the fragment (/path#fragment). This fragment was invisible to the server (it's not routed), so the only way to refresh or bookmark a dynamic page was via JavaScript.

This has major implications for SEO, and led google to develop a seldom-used "hashbang" schema that required a server-side mapping of dynamic hashes to physical URLs. This was unwieldy and not universal among robots, leading the (false) axiom: "spiders can't crawl ajax content". But the benefits of ajax content are tangible: try using google maps w/o JS for example.

The solution was a way to update the URL bar with a value that can be mirrored on the server (allowing bookmarks and JS-less refreshing), WITHOUT reloading the page. Once this capability was available, developers could "navigate" a site by simply updating a "main content section", URL bar, and breadcrumbs. This meant that all the JS+CSS didn't need refetched+parsed, allowing a MUCH faster page-to-page transfer.