How to architecture and expose an API for creating custom data filters?

Question

This question is in the context of a server application - an API that feeds data to its clients.

Here is the obvious: if I have some data, I can search and filter on that data however I please. Let's say I have some data with the following basic structure:

[
  {
    name: 'Paul';
    location: [51.6689134, -0.15818313],
    age: 29,
    gender: m,
  },
  {
    name: 'Emma';
    location: [52.53499242, 0.75317249],
    age: 34,
    gender: f,
  },
  {
    name: 'James';
    location: [50.8162756, -3.08500886],
    age: 27,
    gender: m,
  },
  {
    name: 'Alison';
    location: [48.80648622, -0.60906778],
    age: 41,
    gender: f,
  },
  ...
]

Given that I own the server-side code, I can implement filter options in my API based on any of these fields, for example to fetch only females, or above 40s, or whatever, no problem.

However, here comes the challenge: how can I expose an API to allow developers to create custom filters using my data? For example, someone might want to create a filter to only fetch records of people whose name is exactly 4 characters long, or whose location is within a 1-mile radius of point X - basically, anything goes. I won't know what kinds of creative filters people will come up with. Given that I expect these filters to be created by developers, their implementation may require some programming. However, they (or I) must be able to make these filters public so that average users can use them too - very much like a plugin system.

Here's an example scenario to help with the above description:

1. I have a directory of people, with the above data structure.
2. I have a website where people can search my data based on some basic filters that I provide out of the box.
3. I expose an API, that allows developers to create their own custom filters.
4. A developer uses my API to implement a filter to match person records where the name of the person is N characters long (N being a parameter)
5. A regular user comes to my website, and uses the custom filter that was created by the developer, to look for people whose name is exactly 4 characters long.

EDIT 1: custom filters may have external dependencies, e.g. one might need to use an external API to determine a name's origin, in order to find people with Italian names.

Two things that are important:

1. Performance and efficiency of the system
2. Ease of use for the regular users

From an architectural point of view, how would you implement such a system?

---

EDIT 2

I do understand that this is not a trivial challenge, and that there might not even be a solution to it, due to technical limitations. However, I also believe that there might be a smart solution lurking in the shadows.

Just to clarify, the dataset may have a maximum of 2-3 million records (and it would be nice to allow custom filters at that level), but if necessary, I think I'll be able to reduce the searchable dataset significantly to a maximum of around 1000 entries, before hitting any custom filters.

Answer 1

If you think about it, all that data consist (at lowest level) of some standard primitive types (eg. location is just set of two doubles with arbitrary meaning).

I would go with defining set of operators for each of that types. Equal, Less than, more than, not, in range and so on. For string type i would add also regexp operator (that would fulfill that n-char long and similar requirements). Also grouping and logical operators are needed.

That should be enough for most cases. Of course it would not be possible to do more complex calculations (levenshtein distance for example) but you can add them as operators if and when users need them (remember that their can do further calculations on their side with data grabbed from your API which is perfectly fine).

Other than that is writing good sandbox for execution of foreign, untrusted code inside your application domain. Then your developers can run any code they wish on that data. And here is main problem: how to sandbox that code to be harmless.

You can go with clearing all given code from "bad" things (something like method whitelisting) but that's a lot of work and security holes will always be found.

Another option is to introduce own Turing-complete language that will be compiled on your side to target execution code. Lot of work for you and users (they need to learn it).

Easiest thing to do is to let developers run native code but introduce review process - they send you routine's source code, you review it and if it's harmless then include that in filters set.