0

Im building a project that stores time series data on a per user basis. On the dashboard of the user it'll show some simple statistical analysis like averages but more importantly, it'll create charts based on the data. These charts could be line charts plotting values from one date to another, or a pie chart showing the occurrence/distribution of things.

I've never worked with big data and data that needs to be manipulated so I have a few design questions.

  1. Should I grab all the data at once, or grab the data for a certain time frame one at a time? What I mean is this. Users can view their statistics over the past day, past week, month, whatever. So one thing I can do is grab ALL the data belonging to the user as soon they log in and store it. Then when they ask to see a line chart of their data over the past week, I iterate over the data, pick the points from the past week, display. Then if they want data over the past year, just iterate over the data and so on. The second way is to just query as is needed. So when they load it'll show weekly data as a default. Then if they ask for month it'll requery the DB and get data for the past month. Option 1 is good because it saves having to query the DB every time by storing ALL the data. Option 2 is good since it'll be much less work since I can use SQL to get the exact data I want and display it easily.

  2. Once the data is available to me after I get it from the DB I have two options. I model the data server side and send it properly labeled and ready to be plotted. Or I can send all the data to the front end and I can use JavaScript to model the data client side. With the first way it'll save server resources because the same server is getting hit hundreds or thousands of times a second since it's recording the time series data as well. I'm not sure if this will be an issue since like I mentioned, I've never built anything this big. If it's not an issue for the server to keep getting hit a lot every second (through websockets) and to do data manipulation and everything else, then this isn't a problem and I'd rather do it all server side so I don't expose raw data to the client (not that it's an issue, but still).

I'm using NodeJS for the back end and postgres (timescale) for the DB

0

1 Answer 1

2

Your Client

  • What processing power do they have?
  • What network capacity do they have?
  • What local storage do they have?

Pick the client that has the least of all of the above. That is your common denominator, make sure you have that common denominator for testing with.

Your Data

  • How much data do you need to display the appropriate graphs?
  • How complex is the data?
  • How authoritative does the data need to be?
  • Are there any proprietary algorithms applied to the data to generate the graphs?

Now you can start to answer your question.


Generally speaking offloading the data onto the client will (eventually) allow the client to respond quickly without taxing your server resources, but it will tax the network resources as lots of raw data has to be sent across the network.

It will tax the client resources significantly, as someone has to perform that work, and the client (say a smart watch) may not be up to the task. You also need to consider user behaviour. The use their phone, swap to their tablet, use their laptop, back to their phone. Each change will demand network resources, and also be jarring as the obviously lower powered devices cannot keep up with the user.

One significant pro, is that the user can go offline.

On the otherhand, You can perform all the process on the server and send condensed graphs to the client. You will obviously have to scale your back end to accommodate the extra demand, but it will present a similar experience across all devices. Though some network lag will always be present when drilling down, or otherwise interacting with the data/graphs.


Happy Medium

You want to track for both.

On the devices capable of caching and processing large amounts of data, you want to offload processing to the local device. Perhaps by streaming it, having the device initially use pre-digested graphs from the server, but slowly rely on its own data cache as it warms up.

On devices that don't have the resources, the only way to provide a similar experience is to do the work on the server, and update the clients display with the results.

The only bit you can't shift off your own server would be a proprietary algorithm for munging data. The moment you offload that to a client, its available to the world.

A good analogy would be a map application. On my phone i've downloaded the local maps for my country, I can use them even with mobile data turned off. But when i want to explore another country, i would need to download that regions map first, but then it would be quick and usable offline too.

Via the internet though i can access maps from all over the world, at the level of detail i'm browsing at. When i zoom in the client literally stretches the image it has, but slowly loads extra details in. When i zoom out, it squeezes the image into a thumbnail, and slowly the surrounding area loads, and the thumbnail itself is updated with something prettier at this scale. If i zoom back in sometimes it has to reload the detail view, sometimes it has it in cache. But going offline causes everything to break.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.