Making internal search engine return results that adhere to local permissions on an internal web application

Question

I have an ElasticSearch search engine and an internal web application. Prior to applying the security model, search results were very fast. The security model restricts some users from accessing certain pages. We have 7 "types" of pages.

Our process is currently this:

1. Search engine queries all records
2. For each type of result, I get all recordIds of that type from the database that the user is allowed to access. This ensures I only call the database a max of 7 times (once per type)
3. I do an intersect between the records the user can access and the records search is returning

The above approach isn't fast enough. I have an autosuggest search and each query takes up to 2 seconds now.

What is a typical way to make a search engine return results that adhere to security? Another developer has recommended I store the security permissions directly in ElasticSearch itself(essentially mirroring what our sql server has).

Edit: Each record has a page. We can assume 200-500 users and 25000+ records.

Answer 1

If I understand correctly, previously the search results from the engine was fast. Now you do that, but additionally filter the results afterwards with up to 7 DB queries and intersection filters.

Obviously, the problem lies with the DB queries, or the intersection sorting and filtering!

The next question is - which is slow? Is the DB a slow box, are you performing the filtering once for each DB query? There are usually many ways of optimizing this kind of task, execution plans are your friend in this case.

If the problem is the DB, then caching might be an answer - if the security policies are relatively static, you can fetch the filter restrictions once and save them per user when the user logs in. If the filtering is slow on the DB, then try applying the filter on the web server, a hashmap lookup as each page from the original results can be very fast indeed.

Answer 2

I'm not entirely sure I've understood correctly, but I think you can reduce your 7 database accesses to 1.

1. Encode each security level as a bit in an integer. Store the security level with each record.
2. Establish which security levels a user is allowed to see - encode this as a similar integer.
   1. When searching for records, include a requirement that the bitwise AND of these two values is non-zero.

If the security levels are strictly hierarchical, (e.g. TOP SECRET, SECRET, CONFIDENTIAL, RESTRICTED, UNCLASSIFIED) then this can be simplified further by assigning TOP SECRET = 4, SECRET = 3, CONFIDENTIAL = 2, RESTRICTED = 1, UNCLASSIFIED = 0. Store this with the record as a fieldsecurity_level

Then a user with CONFIDENTIAL clearance can request results with security_level <= 2, etc.

Answer 3

A practice I use for implementing security in search scenario's is like this. You would define groups of users and use these groups to define access to records. When building the index for search, include the access for each group in the index (basically this is the ACL for your record).

You should then have a repository that stores what users are members of what groups. You can then easily retrieve groupmemberships for a user, for example at login.

At query time you would only include the group memberships in your query so that it needs to have at least one of the groups the user is a member of.

In practice the repository holding the group memberships tends to be more complex, because of recursive group memberships.

To summarize:

1. Work with groups
2. Have a security info repository
3. Index your ACL's
4. Query with users group memberships

HTH.