What's actually wrong with an endpoint returning HTML rather than JSON data?

Question

When I first started learning PHP (about 5 or 6 years ago) I learned about Ajax, and I went through "the phases":

1. Your server returns HTML data and you put it inside a DOM's innerHTML
2. You learn about data transfer formats such as XML (and say "oooh so THAT'S what it's used for) and then JSON.
3. You return JSON and build your UI using vanilla JavaScript code
4. You move to jQuery
5. You learn about APIs, headers, HTTP status codes, REST, CORS and Bootstrap
6. You learn SPA, and frontend frameworks (React, Vue.js, and AngularJS) and the JSON API standard.
7. You receive some enterprise legacy code and upon inspecting it, find that they do what's described in step 1.

As I worked with this legacy codebase, I didn't even consider that it could return HTML (I mean, we're professionals now, right?), so I had a hard time looking for the JSON endpoint that was returning the data that the Ajax calls populate. It was not until I asked "the programmer" that he told me it was returning HTML and being appended directly to the DOM with innerHTML.

Of course, this was hard to accept. I started thinking of ways to refactor this into JSON endpoints, thinking about unit testing the endpoints and so on. However, this codebase has no tests. Not a single one. And it's over 200k lines. Of course one of my tasks includes proposing approaches for testing the whole thing, but at the moment we're not tackling that yet.

So I'm nowhere, in a corner, wondering: if we have no tests whatsoever, so we have no particular reason to create this JSON endpoint (since it's not "reusable": it literally returns data that only fits on that part of the application, but I think this was already implied since it... returns HTML data).

What exactly is wrong with doing this?

Answer 1

What's actually wrong with an endpoint returning HTML rather than JSON data?

Nothing, really. Each application has different requirements, and it may be that your application wasn't designed to be a SPA.

It may be that these beautiful frameworks that you cited (Angular, Vue, React, etc...) weren't available at development time, or weren't "approved" enterprisey thingies to be used in your organization.

I'm gonna tell you this: an endpoint that returns HTML reduces your dependency on JavaScript libraries and reduces the load on the user's browser since it won't need to interpret/execute JS code to create DOM objects - the HTML is already there, it's just a matter of parsing the elements and rendering them. Of course, this means we're talking about a reasonable amount of data. 10 megabytes of HTML data isn't reasonable.

But since there's nothing wrong with returning HTML, what you are losing by not using JSON/XML is basically the possibility of using your endpoint as an API. And here lies the biggest question: does it really need to be an API?

Related: Is it OK to return HTML from a JSON API?

Answer 2

JSON and HTML fulfill two different semantic purposes.

If you are populating a web page with data, use JSON. If you are building up a web page from portions of web pages, use HTML.

They may kinda sound like they're the same thing, but they aren't, at all. For one thing, when you are building up a portion of a web page using HTML returned by the server, you are working server-side. When you are binding data to a web page, you are working client-side.

Also, you have to be careful with HTML to not tightly-bind to a specific page. The whole point of rendering partial pages in this way is for the partials to be reusable, and if you make the partial too specific, it won't compose to other pages. JSON doesn't have this problem, because it's just data, not web page structure.

Answer 3

The main problem is that it tightly couples the server to the client, who must know the HTML structure. It also makes the endpoints more difficult to re-use in new ways or new applications.

Returning plain data and letting the client render it decreases coupling and increases flexibility and testability- you can run unit tests on the client for mock data, and run unit tests on the server to test the desired data.

Answer 4

I think you have it a little backwards. You say:

we have no test whatsoever, so we have no particular reason to create this JSON endpoint

A reason to use a proper endpoint would be so that you could test it. I'd say that having no tests is a very good reason to start writing some. That is, if there is any logic that would be suitable to test.

200k lines of code is a lot to refactor and are probably hard to maintain. Breaking out some endpoints and testing them might be a good place to start.

Another reason might be to separate the server from the client. If, in the distant future, the application design or layout changes, it's easier to work with a proper data format than HTML output. In an ideal world, you would only have to change the client and not touch the server at all.

Answer 5

There are 3 ways (at least?) to build up a web page:

• Generate the entire page server side.
• Return a bare bones page from the server plus code (JavaScript), and have the page fetch its data and render into HTML client side.
• Return a partial page plus code, and have the code fetch pre-rendered blocks of HTML that it can drop into the page.

The first one is fine. The second one is also fine. It's the last one that's the problem.

The reason is simple: you have now divided the construction of the HTML page into completely disconnected parts. The problem is one of maintenance. Now you have two separate entities in charge of managing the details of the UI. So you have to keep CSS and other similar details in sync between the two separate pieces. You changed the width of the side bar? Great. Now the HTML fragment that comes back causes horizontal scrolling because its assumptions about the side bar width no longer hold. You changed background color for that block? Great, now your HTML fragment's font color clashes because it assumed a different background color and someone forgot to test that endpoint.

The point is that you have now split up knowledge that should be centralized in a single place (namely the presentation logic), and this makes it more difficult to make sure all the pieces fit together correctly. By using a JSON API, you can instead keep all that logic in just the front end, or you can keep it all in your server side templates if you render your data into HTML to begin with. It's about keeping the presentation knowledge/logic in a single place, so it can be managed consistently and as part of a single process. HTML/CSS/JS is difficult enough to keep straight without breaking it up into a lot of tiny pieces.

JSON APIs also have the additional benefit of making the data available completely independently from the presentation logic. This allows multiple, different presenters, such as both a mobile app and a web page, to consume the same data. In particular, it enables consuming the data without a browser (such as mobile apps or nightly cron jobs); these consumers may not even be able to parse HTML. (This of course necessarily relies on having a situation where the data is the same between the different consumers, or one can use a subset of the other.) Whether you need this ability depends on your particular application's requirements, though, while managing your presentation logic is necessary regardless. I will say that if you implement it up front, though, you'll be better prepared for future growth.

Answer 6

I would say there is nothing wrong with the server returning an HTML fragment and the UI assigning it to .innerHTML of some element. This is, in my opinion, the easiest way to develop asynchronous JavaScript code. The benefit is that as little as possible is done using JavaScript and as much as possible is done in a controlled back-end environment. Remember that JavaScript support in browsers varies but your back-end always has the same version of the back-end components, meaning that doing as much as possible in the back-end will mean as little as possible version incompatibilities.

Now, sometimes you want more than just an HTML fragment. For example, a status code and an HTML fragment. Then you can use a JSON object that has two members, statusCode and HTML of which the second can be assigned to .innerHTML of some element after checking the statusCode. So, using JSON and using innerHTML are by no means alternative exclusive approaches; they can be used together.

By using JSON you can even have multiple HTML fragments in the same response which get assigned to the .innerHTML of multiple elements.

In summary: do use .innerHTML. It makes your code compatible with as many browser versions as possible. If needing more, use JSON and .innerHTML together. Avoid XML.

Answer 7

There is nothing wrong in principle. The question is: What do you want to achieve?

JSON is perfect for transmitting data. If you send HTML instead and expect the client to extract the data from the HTML, that's rubbish.

On the other hand, if you want to transmit HMTL that is going to be rendered as HTML, then send it as HTML - instead of packing the HTML into a string, turning the string into JSON, transmitting JSON, decoding it on the other side, getting a string, and extracting the HTML from the string.

And just yesterday I ran into code that put two items into an array, turned the array into JSON, put the JSON into a string, put the string inside an array, turned the whole array into JSON, sent it to the client, which decoded the JSON, got an array containing a string, took the string, extracted the JSON from the string, decoded the JSON, and got an array with two items. Don't do that.

Answer 8

This all depends on the purpose of the API, but generally what you describe is a pretty strong violation of separation of concerns:

In a modern application, API code should be responsible for data, and client code should be responsible for presentation.

When your API returns HTML, you are tightly coupling your data and presentation. When the API returns HTML, the only thing you can do (easily) with that HTML is display it as some part of a larger page. From a different angle, the only thing the API is good for is supplying your page with HTML. Plus, you've spread your HTML across both the client and server code. This can make maintenance a headache.

If your API returns JSON, or some other form of pure data, it becomes much more useful. The existing app can still consume that data, and present it appropriately. Now, though, other things can use the API to access the same data. Again, too, maintenance is easier - all the HTML resides in one place, so if you want to re-style the whole site, you have no need to change your API.

Answer 9

HTML is tied to a specific design and use.

With HTML, if you want to change the page layout you need to change how the html is generated by the server call. Usually, that requires a back-end programmer. Now you have back-end programmers, who by definition aren't your best html writers, handling these updates.

With JSON, if the page layout changes, the existing JSON server call doesn't necessarily have to change at all. Instead, your front-end developer, or even the designer, updates the template to produce the different HTML you want from the same basic data.

Additionally, the JSON can become the basis for other services. You might have different roles that need to see the same basic data in different ways. For example, you may have a customer web site that shows data about a product in an order page, and an inside sales page for reps that shows the same data in a very different layout, perhaps alongside some other information not available to general customers. With JSON, the same server call can be used in both views.

Finally, JSON can scale better. In recent years we've kind of gone overboard with client-side javascript frameworks. I think it's time to actually take a step back and start thinking about what javascript we're using, and how it affects browser performance... especially on mobile devices. That said, if you're running a site large enough to require a server farm or cluster, instead of a single server, JSON can scale better. Users will give you processing time in their browsers for free, and if you take advantage of that you can reduce the server load in a large deployment. JSON also uses less bandwidth, so again, if you're big enough and use it appropriately, JSON is measurably cheaper. Of course, it can also scale worse, if you end up feeding 40KB libraries to parse 2KB of data into 7KB of html, so again: it pays to be aware of what you're doing. But the potential is there for JSON to improve performance and costs.

Answer 10

There is nothing wrong with such an endpoint if it fulfills its requirements. If it is required to spit out html that a known consumer can parse effectively, sure, why not?

The problem is that, for the general case, you want your endpoints to spit out a payload that is well-formed and effectively parse-able by a standard parser. And by effectively parse-able, I mean, parse-able declaratively.

If your client is forced to read the payload and pry open information bits from it with loops and if-statements, then it is not effectively parse-able. And HTML, being the way its, it is very forgiven in not requiring itself to be well-formed.

Now, if you make sure your html is xml-compliant, then you are gold.

With that said, I have a significant problem with this:

I'm gonna tell you this: an endpoint that returns HTML reduces your dependency on JavaScript libraries and reduces the load on the user browser since it won't need to interpret/execute JS code to create DOM objects - the HTML is already there, it's just a matter of parsing the elements and rendering them. Of course, this means we're talking about a reasonable amount of data. 10 megabytes of HTML data isn't reasonable.

That's a bad idea no matter how you cut it. Decades of collective industrial experience has shown us that, in general, it is a good idea to separate data (or model) from its display (or view.)

Here you are conflating the two for the purpose of speedy JS code execution. And that's a micro optimization.

I've never seen this as a good idea except in very trivial systems.

My advise? Don't do it. HC SVNT DRACONES, YMMV, etc.

Answer 11

JSON is just a textual presentation of structured data. A client naturally needs to have a parser to process data but virtually all languages have JSON parser functions. It is much more efficient to use JSON parser than to use HTML parser. It takes little footprint. Not so with an HTML parser.

In PHP, you just use json_encode($data) and it's up to the client on the other side to parse it. And when you fetch JSON data from a web service, you just use $data=json_decode($response) and you can decide how to use data like you would do with variables.

Suppose you develop an app for a mobile device, why do you need the HTML format when mobile apps rarely use the web browser to parse data? Many mobile apps use JSON (most common format) to exchange data.

Considering that mobiles are often on metered plans, why do you want to use HTML that takes much more bandwidth than JSON?

Why use HMTL when HTML is limited in its vocabulary and JSON can define data? {"person_name":"Jeff Doe"} is more informative than HTML can provide about its data since it only defines structure for HTML parsers, not define data.

JSON has nothing to do with HTTP. You can put JSON in a file. You can use it for configurations. Composer uses JSON. You can use it to save simple variables to files as well.

Answer 12

It's difficult to categorize a right or a wrong. IMO, the questions I'll ask are: "should it", or "can it do with less?".

Every endpoint should strive for communicating with as little content as possible. The signal-to-noise ratio is typically HTTP codes < JSON < XHTML. In most of the situations, it's good to chose the least noisy protocol.

I differ on the point about client browser load by @machado, as with modern browsers this is a non-issue. Most of them are equipped in dealing with HTTP codes and JSON responses pretty well. And although you don't have tests at the moment, long term maintenance of a less noisy protocol would be cheaper than the one above it.