9

I am working on a web app that must deal with very large impulses of simultaneous users, who need to be authorized, to request identical content. In its current state, it's totally crippling to even a 32-core AWS instance.

(Note that we're using Nginx as a reverse proxy)

The response cannot be simply cached since, in the worst case, we must check if the user is authenticated by decoding their JWT. This requires us firing up Laravel 4, which most would agree, is slow, even with PHP-FPM and OpCache enabled. This is mostly due to the hefty bootstrapping phase.

One might ask the question "Why did you use PHP and Laravel in the first place if you knew this was going to be an issue?" - but it's too late now to go back on that decision!

Possible Solution

One solution that has been put forward is to extract the Auth module from Laravel to a lightweight external module (written in something fast like C) whose responsibility it is to decode the JWT and decide if the user is authenticated.

The flow of a request would be:

  1. Check if cache hit (if not pass to PHP as normal)
  2. Decode the token
  3. Check if it's valid
  4. If valid, serve from cache
  5. If invalid, tell Nginx, and then Nginx will then pass the request to PHP to deal with as normal.

This will allow us to not hit PHP once we have served this request to a single user and instead reach out to a lightweight module to mess around with decoding JWTs and any other caveats that come with this type of auth.

I was even thinking of writing this code directly as an Nginx HTTP extension module.

Concerns

My concern is that I've never seen this done before and wondered if there's a better way.

Also, the second you add any user specific content to the page, it totally kills this method.

Is there another simpler solution available directly in Nginx? Or would we have to use something more specialized like Varnish?

My Questions:

Does the above solution make sense?

How is this normally approached?

Is there a better way to achieve a similar or better performance gain?

  • I'm grappling with a similar problem. A couple of ideas a) Nginx auth_request may be able to hand off to your authentication microservice, alleviating the need to develop an Nginx module. b) Alternatively, your microservice could redirect authenticated users to a temporary URL which is public, cacheable and unguessable, but can be validated by the PHP backend to be valid for a limited period (the cache period). This sacrifices some security, if the temporary URL is leaked to a untrusted user, they can access the content for that limited period, much like an OAuth Bearer Token. – James Feb 8 '16 at 2:16
  • Did you come up with a solution to this? I'm facing the same thing – timbroder Dec 9 '16 at 22:40
  • Turns out that by having a large cluster of optimised backend nodes, we were able to deal with the load - but I have high confidence in this approach being a large cost saving solution long-term. If you know some of the responses you could be serving in advance, if you warm the cache before the influx of requests, the backend resource saving and reliability gain would be very high. – iamyojimbo Dec 10 '16 at 0:32
9

I've been trying to address a similar issue. My users need to be authenticated for every request they make. I've been focusing on getting the users authenticated at least once by the backend app (validation of the JWT token), but after that, I decided I shouldn't need the backend anymore.

I chose to avoid requiring any Nginx plugin that is not included by default. Otherwise you can check nginx-jwt or Lua scripting and these would probably be great solutions.

Addressing authentication

So far I've done the following:

  • Delegated the authentication to Nginx using auth_request. This calls an internal location that passes the request to my backend token validation endpoint. This alone doesn't address the issue of handling a high number of validations at all.

  • The result of the token validation is cached using a proxy_cache_key "$cookie_token"; directive. Upon successful token validation, the backend adds a Cache-Control directive that tells Nginx to only cache the token for up to 5 minutes. At this point, any auth token validated once is in the cache, subsequent requests from the same user/token don't touch the auth backend anymore!

  • To protect my backend app against potential flooding by invalid tokens, I also cache refused validations, when my backend endpoint returns 401. These ones are only cached for a short duration to avoid potentially filling the Nginx cache with such requests.

I've added a couple of additional improvements such as a logout endpoint that invalidates a token by returning 401 (which is also cached by Nginx) so that if the user clicks logout, the token cannot be used anymore even if it's not expired.

Also, my Nginx cache contains for every token, the associated user as a JSON object, which saves me from fetching it from the DB if I need this information ; and also saves me from decrypting the token.

About token lifetime and refresh tokens

After 5 minutes, the token will have expired in the cache, so the backend will be queried again. This is to ensure that you're able to invalidate a token, because the user logs out, because it has been compromised, and so on. Such periodic revalidation, with proper implementation in the backend, avoids me to have to use refresh tokens.

Traditionally refresh tokens would be used to request a new access token; they would be stored in your backend and you would verify that a request for an access token is made with a refresh token that matches the one you have in the database for this specific user. If the user logs out, or tokens are compromised you would delete/invalidate the refresh token in your DB so that the next request for a new token using the invalidated refresh token would fail.

In short, refresh tokens usually have a long validity and are always checked against the backend. They are used to generate access tokens that have a very short validity (a few minutes). These access tokens normally do reach your backend but you only check their signature and expiration date.

Here in my setup, we are using tokens with a longer validity (can be hours or a day), that have the same role and features as both an access token and a refresh token. Because we have their validation and invalidation cached by Nginx, they are only fully verified by the backend once every 5 minutes. So we keep the benefit of using refresh tokens (be able to quickly invalidate a token) without the added complexity. And simple validation never ever reaches your backend that is at least 1 order of magnitude slower than the Nginx cache, even if used only for signature and expiry date checking.

With this setup, I could disable authentication in my backend, since all incoming requests reach the auth_request Nginx directive before touching it.

That doesn't fully solve the problem if you need to perform any kind per-resource authorization, but at least you've saved the basic authorization part. And you can even avoid decrypting the token or do a DB lookup to access token data since Nginx cached auth response can contain data and pass it back to the backend.

Now, my biggest concern is that I may be breaking something obvious related to security without realizing it. That being said, any received token is still validated at least once before being cached by Nginx. Any tempered token would be different so wouldn't hit the cache since the cache key would also be different.

Also, maybe it's worth mentioning that a real world authentication would fight against token stealing by generating (and verifying) an additional Nonce or something.

Here is a simplified extract of my Nginx config for my app:

# Cache for internal auth checks
proxy_cache_path /usr/local/var/nginx/cache/auth levels=1:2 keys_zone=auth_cache:10m max_size=128m inactive=10m use_temp_path=off;
# Cache for content
proxy_cache_path /usr/local/var/nginx/cache/resx levels=1:2 keys_zone=content_cache:16m max_size=128m inactive=5m use_temp_path=off;
server {
    listen 443 ssl http2;
    server_name ........;

    include /usr/local/etc/nginx/include-auth-internal.conf;

    location /api/v1 {
        # Auth magic happens here
        auth_request         /auth;
        auth_request_set     $user $upstream_http_X_User_Id;
        auth_request_set     $customer $upstream_http_X_Customer_Id;
        auth_request_set     $permissions $upstream_http_X_Permissions;

        # The backend app, once Nginx has performed internal auth.
        proxy_pass           http://127.0.0.1:5000;
        proxy_set_header     X-User-Id $user;
        proxy_set_header     X-Customer-Id $customer;
        proxy_set_header     X-Permissions $permissions;

        # Cache content
        proxy_cache          content_cache;
        proxy_cache_key      "$request_method-$request_uri";
    }
    location /api/v1/Logout {
        auth_request         /auth/logout;
    }

}

Now, here's the configuration extract for the internal /auth endpoint, included above as /usr/local/etc/nginx/include-auth-internal.conf:

# Called before every request to backend
location = /auth {
    internal;
    proxy_cache             auth_cache;
    proxy_cache_methods     GET HEAD POST;
    proxy_cache_key         "$cookie_token";
    # Valid tokens cache duration is set by backend returning a properly set Cache-Control header
    # Invalid tokens are shortly cached to protect backend but not flood Nginx cache
    proxy_cache_valid       401 30s;
    # Valid tokens are cached for 5 minutes so we can get the backend to re-validate them from time to time
    proxy_cache_valid       200 5m;
    proxy_pass              http://127.0.0.1:1234/auth/_Internal;
    proxy_set_header        Host ........;
    proxy_pass_request_body off;
    proxy_set_header        Content-Length "";
    proxy_set_header        Accept application/json;
}

# To invalidate a not expired token, use a specific backend endpoint.
# Then we cache the token invalid/401 response itself.
location = /auth/logout {
    internal;
    proxy_cache             auth_cache;
    proxy_cache_key         "$cookie_token";
    # Proper caching duration (> token expire date) set by backend, which will override below default duration
    proxy_cache_valid       401 30m;
    # A Logout requests forces a cache refresh in order to store a 401 where there was previously a valid authorization
    proxy_cache_bypass      1;

    # This backend endpoint always returns 401, with a cache header set to the expire date of the token
    proxy_pass              http://127.0.0.1:1234/auth/_Internal/Logout;
    proxy_set_header        Host ........;
    proxy_pass_request_body off;
}

.

Addressing content serving

Now the authentication is separated from the data. Since you told it was identical for every user, the content itself can also be cached by Nginx (in my example, in the content_cache zone).

Scalability

This scenario works great out of the box assuming that you have one Nginx server. In a real world scenario you probably have high availability, meaning multiple Nginx instances, potentially also hosting your (Laravel) backend application. In that case, any request your users make could be sent to any of your Nginx servers, and until they all have locally cached the token, they will keep reaching your backend to verify it. For a small number of servers, using this solution would still bring big benefits.

However, it is important to note that with multiple Nginx servers (and thus caches) you lose the ability to log out on the server side because you're unable to purge (by forcing a refresh) the tokens cache on all of them, like /auth/logout does in my example. You're only left with the 5mn token cache duration that will force your backend to be queried soon, and will tell Nginx that the request is denied. A partial workaround is to delete the token header or cookie on the client when logging out.

Any comment would be very welcomed and appreciated!

  • You should be getting a lot more upvotes! Very helpful, thanks! – Gershon Papi Jun 10 at 22:01
  • "I've added a couple of additional improvements such as a logout endpoint that invalidates a token by returning 401 (which is also cached by Nginx) so that if the user clicks logout, the token cannot be used anymore even if it's not expired." - This is clever! , but are you actually blacklisting the token in your backend as well, so that in case the cache gets down or something, the user is still not able to login with that particular token? – gaurav5430 Jun 22 at 15:42
  • "However, it is important to note that with multiple Nginx servers (and thus caches) you lose the ability to log out on the server side because you're unable to purge (by forcing a refresh) the tokens cache on all of them, like /auth/logout does in my example." can you elaborate? – gaurav5430 Jun 22 at 15:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.