0

I have started learning streaming APIs and I found one of the good documentations here. There was a comparison given by author to demonstrate the effectiveness of the streams.

var http = require('http');
var fs = require('fs');

var server = http.createServer(function (req, res) {
    fs.readFile(__dirname + '/data.txt', function (err, data) {
        res.end(data);
    });
});
server.listen(8000);

It is told that in the example above, for every request the whole file would be read and stored in memory which may create problems for large number of concurrent connections. Sounds good!

Now, for the solution in the second example:

var http = require('http');
var fs = require('fs');

var server = http.createServer(function (req, res) {
    var stream = fs.createReadStream(__dirname + '/data.txt');
    stream.pipe(res);
});
server.listen(8000);

it is told that since res is itself is a stream hence we can read the file via stream and pipe the result into res which would not create the memory issue.

Question

Will the browser would keep the connection open util the whole file is read since according to me the browser knows about HTTP only hence how would it handle the stream scenario. Also, wouldn't it take longer to stream the file vs sending whole file at a time?

0

wouldn't it take longer to stream the file vs sending whole file at a time?

Welcome to the time space tradeoff. Reading a whole file into memory (say as an array or string) is called slurping. It can be a good or bad idea depending on file size, available memory, and how many times you do it at the same time.

The alternative is line by line processing (or chunk processing for binary files). As smaller parts of the file come in they are acted on and then forgotten. This keeps the memory footprint small but increases accessing overhead. That's the tradeoff that you must balance.

Streams can do either. When you offer access as a stream you're saying, "I don't know which you're about to do but here's where to get your data". Once you have a stream you can dump it all in memory, act on smaller parts, or hand it to something else as a stream.

  • Ehm, I think only the Perl folks call it "slurping." – Robert Harvey Jul 10 '16 at 15:00
  • 1
    @CandiedOrange: Please don't tell people that "slurping" is a real term. It's a word some Perl nut made up when they were trying to be cute. I've been in this industry a long time, and "slurping" has never been a thing, unless you're referring to this. Nobody who wants to be taken seriously goes into a code review and declares "I wrote this code to slurp a file." – Robert Harvey Jul 10 '16 at 20:29
  • 1
    @robertharvey I've been in this industry since it was only 30 years old. None of the terms are "real". We beg, borrow, and steal terms from other industries let alone other programming languages. If you don't want a nomenclature debate then delete ALL your nomenclature debate comments. If you insist on having one please make a better argument than name calling. – candied_orange Jul 10 '16 at 21:27
  • 1
    If you're actually one of those perl nuts, then I apologize for that. Don't let it diminish my central point, though. Young programmers have enough problems distinguishing terms from concepts without us throwing fictional ones at them. Surely we have enough mental masturbation over SOLID and software patterns already. – Robert Harvey Jul 10 '16 at 23:23
  • 1
    @stakx I appreciate the support. It's true that it doesn't need any special term. Just as, "in my humble opinion" doesn't need any special abbreviation. Yet here we are. – candied_orange Jul 15 '16 at 6:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.