I am developing an iOS application that plots a large number of locations on a map, I have selected a data structure that is suitable for the pins on the map, though I am stuck on an approach for the data.
Each entry in the dataset is an object that contains a few strings primarily identifying items such as street address, lat/long, and location type, so each object in the set isn't large. This data comes from multiple third party APIs.
The issue I am running into comes from the size of the data (up to 150,000 objects) and the constraints I have with the app.
Constraints for the app:
- The app needs to be usable without an internet connection, so the data has to be cached on the device when it is first downloaded.
- I would like to avoid any "redo search in this area" function, meaning the data should easily be browsable by the user without having to wait for further network requests (thus the app needs all the data at some point). This is primarily because it's better from a user experience standpoint, and the fact that all the locations are relatively tightly packed (area is 2000 square kilometres, so there isn't much pan space on a map)
My current approach involves a Node.JS backend that on startup will fetch the data from the various third party APIs and load it into memory to improve response time when a request comes in. The data is cached for 24 hours (data updates daily), after which, the data is re-fetched and cached into memory again.
The app on first boot will display a message/spinner to the user that it's fetching the data, when the data is fetched, it is cached for offline use. Every period of time, the app will attempt to refresh the data from the backend in the background, and if an update is available, the data will be updated accordingly.
Now for the few questions I had:
- Is having a paginated endpoint logical despite the fact that the client will almost always need all the data? Or better, what would be a logical endpoint design given the above use-case? I'm leaning towards some sort of pagination to minimize the size of each individual response to the client, also given that a Heroku deployed app with an endpoint that returned all the objects in one go sometimes took up to 10 seconds to respond.
- Is the overall approach logical given the above information? Or is there something that can be changed in the design/approach of the problem that will yield better performance/results?