I'm new to server development, and I'm trying to figure out where the division of responsibilities is for Data intensive tasks, I'm using Node.js.

As an example say I'm making a Single Page Application:

  • The Client renders a map with population per area
  • The Server handles requests for the population data

However, assume that I'm also not the owner of the data, but that I collect it from a third party via an API. That API simply gives me a list of people and their location. I will have to scrub through each person, group them based on location, total them up and then store that data so it can be accessed for the purpose of the application. This process doesn't need to be done all the time, since population won't be changing all the time, but say I refresh this dataset once a day.

My question is putting this process on the Server seems like it would be process intensive enough that it would slow down requests. I see a few options but I'm not sure which one would be used in a professional development setting/ be most efficient.

I could:

  • Send the client the raw data, and they can do all the parsing themselves - but then the user experience slows down across the board
  • Have the Server do the processing on the same thread, and once a day the server slows down
  • Spawn worker threads on the same server to handle it (discouraged?)
  • Have an entirely separate third server, which my main server pings once a day to process the data and return it.

Is there another way of doing this that I'm not aware of, or is any one of these methods better practice?

  • 3
    CPU-intensive, data-intensive tasks go wherever they most effectively meet your specific requirements. That might be on the same server, it might be on auxiliary servers, or it might be on the client. Depends entirely on the application and its specific characteristics. – Robert Harvey Mar 27 '18 at 17:41
  • 2
    A few things to consider. What devices are your clients using? If they are likely to be using mobile devices or unplugged laptops, putting a lot of client-side computation may hurt their battery life. What would they think of this? Can you use caching to minimize the amount of computation across all clients if you do it on the server? Or at least to possibly minimize requests to third-party services, either to speed up the process, minimize request time, or handle downtime in the third parties? More servers means more opportunities for failure or need for maintenance - can you support this? – Thomas Owens Mar 27 '18 at 17:53
  • 1
    @ThomasOwens There is also an element of control to consider. You have full control over what happens on the server. You can only hope that clients do what you expect. – Eric May 8 '18 at 22:18

The normal rule for data intensive work is to place the processing so that the 'large' block of data doesn't move around much. This is because network delays are frequently much larger than any other single activity delay.

From what you've described though, the answer may depend on the total data table size. If the total population is only a few dozen and will stay small, then transferring all the data to the client and processing it there won't impose a significant delay.

However, it sounds like your problem could easily have a population of hundreds or thousands. In that case you definitely don't want to toss all the raw data across the network in response to every query. Instead, go with your choice to spawn a worker thread on the server. There are several possible strategies to spread out the work of aggregating the population data adequately to maintain server performance at an acceptable level.

| improve this answer | |

You always want to minimise the amount of data transmitted between server and client. If by doing some amount of extra work the server can significantly reduce the amount of data to be sent, then this is most likely a win.

Otherwise, you have the choice of reducing the total amount of work done, or the total cost to you. If you run a server serving 1000 users, who all use modern smart phones, then their total computing power may be ten times or hundred times more than yours. There is a good chance that by moving the hard work to the clients, your server can deliver data to everyone faster, and overall the time the users wait is minimised, plus your cost is minimised.

What would be more programming effort would be to be able to do either (calculate on the server or the client) and use it like load balancing: If your server gets too many requests and starts falling behind, it sends the work to the clients.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.