5

I have two Float32Arrays each one of which is 1.6*10^7 in length(floating point array). Using JS I retrieve them from server and add them element by element. My webpage stops responding and I get the following error message. Is there a way of handling huge arrays on client side without using JS or with JS?

enter image description here

  • 9
    That's 128 freaking megabytes if my calculations are correct, and assuming you send them in perfectly packed binary rather than in JSON or something. It's debatable whether handling this much data on the client side is sensible, but sending them from the server to the client is insane. What do you need 32 million numbers for anyway? – user7043 Aug 13 '16 at 22:10
  • why don't you retrieve it chunk by a chunk from the server, instead of retrieving it at once, get a chunk --> process it --> store the result on the client --> repeat – David Aug 19 '16 at 2:12
  • When you say add them element by element, do you mean using .push() on the array? It almost sounds like you mean adding to the DOM. – joshp Aug 21 '16 at 10:47
  • 1
    What are you doing with these arrays once you retrieve them? I believe it is essential to know more details to guarantee a good answer. – kamoroso94 Aug 22 '16 at 2:38
3

I don't have previleges to comment. So, I am adding it as an answer here.

My browser (firefox 47) was able to set values for 50,000,000 elements of a Float32Array, one by one, under one second (It is not an insert/append though). It ran out of memory beyond that.

I assume your bottleneck (browser warning) has more to do with slow fetching/processing of elements, few at a time, and keep the main browser thread busy.

If you really need that much data to be fetched/processed/inserted, while at the same time let the browser remain responsive to the user, you may want to consider the multi-threading in HTML5 (or javascript libraries provided by Mozilla and others). It will help you in offloading the busy work into a background thread.

1

Use range requests to get the data from the server chunk by chunk and send it back to the server chunk-by-chunk. An example of using range requests, based on this SO question is shown below:

var chunkSize = 8388608; // for 8MB segments
function getPartialSegment(fileURL, chunkN, processData, whenError, prevreq){
    var xmlhttp = prevreq || new XMLHttpRequest();
    xmlhttp.open("GET", fileURL, true);
    xmlhttp.setRequestHeader(
        "Range",
        "bytes=" + (chunkN*chunkSize) + "-" + ((chunkN+1)*chunkSize - 1)
    );
    xmlhttp.responseType = 'arraybuffer';
    xmlhttp.onload = function(){
        var response = xmlhttp.response;
        if (response && (+xmlhttp.status === 200 || response.length > 0)){
            if (response.byteLength%4){ 
              if (!response.transfer){
              var responseUint8 = new Uint8Array(response),
                 tmpDataBuffer=new ArrayBuffer(Math.floor(response.byteLength/4)*4+4),
                  tmpDataUint8 = new Uint8Array(tmpDataBuffer)
                tmpDataUint8.set( responseUint8 );
                processData(new Float32Array(tmpDataBuffer), xmlhttp)
              } else processData(new Float32Array(ArrayBuffer.transfer(response, 
                Math.floor(response.byteLength/4)*4+4)),xmlhttp)
            } else processData(new Float32Array(response), xmlhttp);
        } else if (whenError instanceof Function)
            whenError(xmlhttp.status, xmlhttp);
    };
    xmlhttp.send( null );
};

However, please note that for this method to work, you will have to set up your servers to accept range requests. Then, based on this other SO post, you can get the remote file size. However, this will require your server to set the content-length header to the file size. So, putting it all together into a small parser, you get something like this (the following snippet of code is fully functional except it requires the getPartialSegment above):

function proccessEntireFileInChunks(fileURL, processData, allDone, whenError){
    var xmlhttp=new XMLHttpRequest();
    xmlhttp.open("HEAD", fileURL, true);
    xmlhttp.onload = function(){
      var filelen = parseInt(xmlhttp.getResponseHeader("Content-Length"))
      if (xmlhttp.status === 200 || !isNaN(filelen)){
        var chunks  = Math.ceil(filelen / chunkSize),
            chunkN  = 0;
        if (!chunks) return allDone && allDone();

        getPartialSegment(fileURL, chunkN, function nextFile(response){
          processData(response, chunkN*chunkSize, filelen, xmlhttp);
          if (++chunkN !== chunks){
            getPartialSegment(fileURL, chunkN, nextFile, whenError, xmlhttp);
          } else return allDone && allDone();
        }, whenError, xmlhttp);
      } else if (whenError instanceof Function)
        whenError(xmlhttp.status, xmlhttp);
    };
    xmlhttp.send( null );
}

Having said all that, I've just got to say that if your only going to add up the elements, then it will tax the server a lot less to add them up on the server instead of on the client. I too am very enthusiastic about moving as much of the processing to the client as possible, but sending data across a network just for addition is a lot less efficient that adding it without sending it across a network. I wouldn't be surprised if internally, it adds a pointer that's used to read/write all of the bytes in the data to the network request meaning that sending it across the network to get added induces 3 extra additions (absolute bare minimal, though likely more in reality) per byte in your array on the server that could have been just a single addition per element on the server.

0

The answer is that you can't process all that stuff toghether, because is too much.

You should rethink about what you are doing, some possible solution are:

  • split the data in chunks
  • reduce the size with filters
  • pre-elaborate on server side
  • ... be creative
0

What you need is processing relative big data in a low memory, low performance environment. The general solution to this is using streams. In these streams you put only a single or a few chunks in the memory, process it, and free the memory. So you won't need a lot of memory and processing power to do the job.

You need to stream the data from the server, create a processing stream on the client and display the data chunk by chunk, probably pixel by pixel. I guess you'll need a zooming algorithm, which should be able to use the same streams.

If you have a working solution, then you can try to make it faster, e.g. by using multiple websocket connections, multiple display streams, a client side storage to cache data, data preparation on server side, web workers, etc... I think this will be a much longer project than you expected... Don't forget to share the solution with the js community! :-)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.