I have a data set of over a million addresses and I want to display the closest N locations to a given address or current location

Question

I am a student working on a personal project which is essentially a location finder that will be on a website. I have a data set of over a million addresses and I want to display the closest N locations to a given address or current location.

I would greatly appreciate some examples of the technologies/stack that an experienced developer might use to approach this problem. I know I need to set up a database to query the locations so I've tried to import the data to a MySQL server but I've been having trouble due to the size of the data set.

From there, I know I need to geocode the database of addresses to get the lat/long values needed to calculate the distances. But would the search algorithm be impractical as it needs to calculate each of over a million entries? Is there a way to optimize that? I was also thinking about a K Nearest Neighbors using a KD Tree with the lat/long values as the axes.

Sorry for the rambling, I have not been involved in a project of this scale and have no idea which technologies I need to use to approach this. I would appreciate some suggestions for how to implement this and some of the various technologies that may be involved. Thank you for your time.

Answer 1

Let us start with

I know I need to set up a database to query the locations so I've tried to import the data to a MySQL server but I've been having trouble due to the size of the data set.

This has not much to do with your question's title, but all major DB systems support a form of "bulk insert". For MySQL, look into this SO question as a starter.

From there, I know I need to geocode the database of addresses to get the lat/long values needed to calculate the distances. But would the search algorithm be impractical as it needs to calculate each of over a million entries?

That is why you should not do this "manually". Use a database with so-called spatial data types and spatial indexes (see, for example, these MySQL docs).

This older SO question shows a complete example for finding nearest points to a given point using spatial data types.