3

To some extent, this is a wide question, because I do not know in which direction I should move. I am using Polymaps to show markers on a map. The markers are static but the visualization depends on the values of the input data. For the map I read the data like this:

    var b = read_function("data.json", function(a) {
        json_object = a
    });

    map.add(b.on("load",load))

This works well when the JSON files are relatively small. When I get above 30 MB in size, the browser starts to complain (slow, crashing etc.). I work with Numerical Weather Prediction (NWP) model data so I am mainly experienced with Fortran and Python and the data I hope to be able to show is for an entire model domain of 1.3 million points, that is 1.3 million markers. The JSON file with all this data takes up 1.1 GB.

Surely it is not wise to read this file at once and I do not want to show all markers at the same time anyway. The density of markers can be much lower when the zoom is low and show more and more as you zoom in.

So my question is, how should an amateur proceed? Do I perhaps need to create an API somehow that returns smaller JSON objects depending on the state of the map?

  • Where does the JSON file come from, and how often does it change? – Doc Brown Jan 31 at 11:14
  • The JSON is generated from model data which covers several years. The data in the file represents climatology, so once created the file will never change. – Whir Jan 31 at 11:21
5

Since your JSON file never changes, it is an ideal candidate for preprocessing it once, extracting the marker positions, and store them in a file format which is optimized for your specific purpose.

You have to solve two different problems here:

  • Create individual sets of markers of different densities for different zoom levels ("level of details"). That should be not too hard, just process the huge JSON file once for each zoom level, decide which markers you want to keep and which to throw away, then write a new JSON file which contains exactly these markers.

    For an optimized solution of this task, use a "poor mans spatial index": lets say you want not more than one marker in every tile of 10x10m size for a certain zoom level. Then round each marker's coordinate pair (X,Y) to the next multiple of 10 meters and use the rounded coordinate pair as the key for a dictionary of markers with one entry per key at most.

  • Especially for zoom levels which require a high level of details, the idea for not having to load and draw the markers all at once is to split up the whole set into tiles of a certain size, and use one file per tile (usually named by the upper left coordinates of the tile). If only a portion of the markers is visible, only the related tiles can be loaded from disk and shown in the browser.

    I am not an expert in Polymaps, but having a short look into the manual it seems they provide support for exactly this kind of tiled data organization. So this is the point where one has to dive into the manual of that lib, start looking at examples like this one and try to implement something similar (and by "one", I mean "you" - ;-) ). After giving this a serious try, in case you run into troubles, I would recommend to ask about the specific coding problem at Stackoverflow.

1

Fortran and Python are already great tools for number crunching, and, if you know you don't want to show all the markers at once, you will have some "filter" rules based on your inputs, right? They have probably streams or buffers that can allow you to read a file by not loading it into memory all at once.

How about you do some crude pre-processing in a way that enables you to partition your JSON markers file in bulks based on inputs or "ranges of inputs"?

I talk about doing this processing "offline", so lets say you roughly know a range of "zoom" values and which points to encompass in each:

You can create a map or associative list like:

  • Zoom level between 1 and 5: JSON([list of corresponding points for this range])
  • Between 6 and 8: JSON([filtered list])
  • Larger than 8: JSON([remaining list])

So if you have zoom 3, you fall on the first category, zoom 8 on the second etc.

The list lookup for the initial bucket to fall into is easy to make and for cases where these lists grow large, if they have a start and end, the lookup takes O(1) only, and then its a matter of working with a pre-processed list.

Maybe this can give you some idea how to proceed.

1

First, I like the @DocBrown overall approach. However, for something possibly simpler, if the data looks something like

[
  {
    point 1 data
    {
       possible with nested stuff BUT without any 
      "some field name{" : " some } data" 
       that include brackets to screw up simple counting of brackets
    } 
  },
  ...
  { 
    point 1,300,000
  }
]

You could write simple parsing code to grab one point inside a { ... } block at a time and parse it. So, at a low zoom level, just grab the first 100 or so and show them. As the user zooms in, read more of the file and show more points. Since you are reading the file just a few lines at a time there shouldn't be any memory issues from the file I/O. (I hope)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for?Browse other questions tagged or ask your own question.