We have a list of about 100, 000 customers. Every night, we run a batch search of these customer names against a list of known criminals/suspicious persons. This list contains around 1 million entries.
To ameliorate false-negatives, we're using fuzzy string matching which catches small mismatches between the customer names and the suspicious names lists.
The problem is that this search is incredibly slow (it might take days to finish).
What strategies can I consider in order to avoid checking customers again and again against some entries?
Note: The suspicious-persons list is updated every week with new entries.
Possible solutions
- I could keep a table that records which customers have been checked against which entries. However this is a terrible solution. I need to keep a record of 100,000 customers * 1 million entries in order to cover all the possibilities.