0

I have an ElasticSearch search engine and an internal web application. Prior to applying the security model, search results were very fast. The security model restricts some users from accessing certain pages. We have 7 "types" of pages.

Our process is currently this:

  1. Search engine queries all records
  2. For each type of result, I get all recordIds of that type from the database that the user is allowed to access. This ensures I only call the database a max of 7 times (once per type)
  3. I do an intersect between the records the user can access and the records search is returning

The above approach isn't fast enough. I have an autosuggest search and each query takes up to 2 seconds now.

What is a typical way to make a search engine return results that adhere to security? Another developer has recommended I store the security permissions directly in ElasticSearch itself(essentially mirroring what our sql server has).

Edit: Each record has a page. We can assume 200-500 users and 25000+ records.

8
  • 1
    I have only practical experience with Solr, but Elasticsearch should be similar enough. I would just tag the documents with the 'pages' information. (Basically as your developer recommended). This would only another type of tag and our queries use several of them for every query anyway without any performance problems. The search engine can handle this internally much faster than your code. (Though in theory you could try to optimize your intersect algorithm, but it would depend on many factors if this is worth your time when letting ES do it will most likely be easy enough) Aug 27 '14 at 14:26
  • 1
    I'm not familiar with your technologies but 7 queries is a lot. Can you change a where x = to where x in and do them all in one query?
    – Daenyth
    Aug 27 '14 at 14:30
  • @thorstenmüller We have ~200-500 users and each user will have permission to view different items. Seems like a lot of info to attach to each record. Right now the permissions are stored in a 2d matrix. Does this change anything, or same recommendation?
    – Brandon
    Aug 27 '14 at 14:30
  • @Daenyth I can, but the query would be more intensive with multiple joins and such. With 7 separate queries, I only run them if I have a result of that type. If I only have a result of type "contact" then I'd only be running one query.
    – Brandon
    Aug 27 '14 at 14:34
  • Not explained very well, but permissions are always for 'pages' did I get this right? So you tag each document with the page it belongs to. Then before the query you get the pages for the user (say [2, 5, 7]) and ask ES to return only documents that have one of these as tag. That's how I understood the problem here. Alternately you could also tag every doc with the user-id of everybody who is allowed to access it, though this would be a bit more complicated (though for about 500 users this should still work at least Solr can handle such things surprisingly performant) Aug 27 '14 at 14:38
1

If I understand correctly, previously the search results from the engine was fast. Now you do that, but additionally filter the results afterwards with up to 7 DB queries and intersection filters.

Obviously, the problem lies with the DB queries, or the intersection sorting and filtering!

The next question is - which is slow? Is the DB a slow box, are you performing the filtering once for each DB query? There are usually many ways of optimizing this kind of task, execution plans are your friend in this case.

If the problem is the DB, then caching might be an answer - if the security policies are relatively static, you can fetch the filter restrictions once and save them per user when the user logs in. If the filtering is slow on the DB, then try applying the filter on the web server, a hashmap lookup as each page from the original results can be very fast indeed.

1

I'm not entirely sure I've understood correctly, but I think you can reduce your 7 database accesses to 1.

  1. Encode each security level as a bit in an integer. Store the security level with each record.
  2. Establish which security levels a user is allowed to see - encode this as a similar integer.
    1. When searching for records, include a requirement that the bitwise AND of these two values is non-zero.

If the security levels are strictly hierarchical, (e.g. TOP SECRET, SECRET, CONFIDENTIAL, RESTRICTED, UNCLASSIFIED) then this can be simplified further by assigning TOP SECRET = 4, SECRET = 3, CONFIDENTIAL = 2, RESTRICTED = 1, UNCLASSIFIED = 0. Store this with the record as a fieldsecurity_level

Then a user with CONFIDENTIAL clearance can request results with security_level <= 2, etc.

0

A practice I use for implementing security in search scenario's is like this. You would define groups of users and use these groups to define access to records. When building the index for search, include the access for each group in the index (basically this is the ACL for your record).

You should then have a repository that stores what users are members of what groups. You can then easily retrieve groupmemberships for a user, for example at login.

At query time you would only include the group memberships in your query so that it needs to have at least one of the groups the user is a member of.

In practice the repository holding the group memberships tends to be more complex, because of recursive group memberships.

To summarize:

  1. Work with groups
  2. Have a security info repository
  3. Index your ACL's
  4. Query with users group memberships

HTH.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.