5

If we have a long list of JSON entries, we could put those on an HTTP body so we could parse the whole body of JSON with something like JSON.parse():

HTTP/1.1 200 OK
Date: Sun, 10 Oct 2010 23:26:07 GMT
Server: Apache/2.2.8 (Ubuntu) mod_ssl/2.2.8 OpenSSL/0.9.8g
Last-Modified: Sun, 26 Sep 2010 22:04:35 GMT
ETag: "45b6-834-49130cc1182c0"
Accept-Ranges: bytes
Content-Length: (big number)
Connection: close
Content-Type: application/json

[
"imagine",
"this",
"list",
"having",
"thousands",
"of",
"entries",
...
]

Alternatively, we could split up the body into new lines.

Is there a way to write the HTTP body in a streaming manner (line by line) and also read the HTTP body in a streaming manner? Or is there no point since the HTTP body is always buffered?

  • my guess is that if you really want streaming ability that you would use pure TCP not HTTP. – Alex Channelmeter Feb 25 at 18:22
  • yes but why realtimeapi.io/hub/http-streaming – Ewan Feb 25 at 18:24
  • 1
    Of course there is a way. – user253751 Feb 25 at 18:34
  • why? for lots of data in a request so it doesn't have to buffer everything I guess – Alex Channelmeter Feb 25 at 19:00
  • 1
    Isn't streaming in this context simply displaying part of the file before you've received all of it? So long as the file format supports that you're fine. Many webpages can be displayed before they're fully loaded. – candied_orange Feb 25 at 19:03
2

Reading a HTTP response as a stream is trivial using most clients. Just request a stream and use a json parser that supports streams. For example, using .NET: https://docs.microsoft.com/en-us/dotnet/api/system.net.http.httpclient.getstreamasync?view=netframework-4.8

Writing a HTTP response as a stream can be a bit more complicated if it's dynamically generated because you don't know the length at the beginning and some frameworks might require some tweaking so that they don't buffer the entire response before sending it.

| improve this answer | |
2

This question seems a little confused. HTTP and HTTP/2 are TCP-based protocols Unless you are experimenting with HTTP/3, all HTTP interactions are pure TCP. There's really nothing special about streaming a response from HTTP/TCP perspective. All the responses are sent as a stream of packets. The challenge with streaming and HTTP is building the server such that it writes responses to the output stream without loading the entire content into memory first. Here's a low-level example of doing that in Java: https://stackoverflow.com/questions/55709/streaming-large-files-in-a-java-servlet

On the other end, the challenge is have a client that supports handling content from the server before the entire stream is read. A standard jquery ajax call will not do this. It will load all the content before firing the 'success' event. An example client-side library that does support this is oboe.js.

One challenge with this approach (and why I think it's somewhat unusual) is that it's a little awkward handling errors that occur once you've started processing a stream. Often it simpler to just use other approaches to breaking up a large response into multiple complete request/responses. For example, if you have a very large JSON document with nested structures, you can create sub-resources for those nested structures and put URIs to those resources in place of the nested structures. This often works well with user flows where the user doesn't want or need to see every part of the response (which is typical for large documents.)

| improve this answer | |
0

HTTP has its own semantics for streaming, please see content-range and chunking.

| improve this answer | |
  • any example of how to send and receive with code would be ideal – Alex Channelmeter Feb 25 at 20:00
  • which language? – Martin K Feb 25 at 20:11
  • node.js, golang or java, any of those would be my preference, thx – Alex Channelmeter Feb 25 at 20:27
0

Server-sent events seem to fit the requirements as-stated.[1] They permit you accept a request server-side exactly as you would a normal request, except you respond with a Content-Type: text/event-stream header and you output event lines to the client as-needed. (And flush.)

Minimal example adapted from github:mdn/dom-examples:

<?php
date_default_timezone_set("America/New_York");
header("Content-Type: text/event-stream");

while (true) {
  $curDate = date(DATE_ISO8601);
  echo "event: ping\n",
       'data: {"time": "' . $curDate . '"}', "\n\n";

  flush();
  if ( connection_aborted() ) break;
  sleep(1);
}

Client-side, you set up an EventSource which receives each "event" individually:

var evtSource = new EventSource('sse.php');
var eventList = document.querySelector('ul');

evtSource.onmessage = function(e) {
  var newElement = document.createElement("li");

  newElement.textContent = "message: " + e.data;
  eventList.appendChild(newElement);
}

(From MDN's page on the EventSource API.)

As the name implies, you'd normally use SSE's for pushing "events" to the client. (E.g., chat messages from a peer.) But, if you're trying to provide quick feedback for slow loading, for example, your "events" could just be record_loaded or something.


1. Notably, my opinions and advice might change if I knew what your doing at a high level...

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.