I've created a small prototype browser plugin and am now thinking about making it accessible to the public. This brings up an important question about gatekeeping API access and the right way to deploy a potentially expensive service.
The plugin uses Firebase for user authentication and a database. It also needs access to a custom machine learning model. I've set up this model in a Docker container which runs a FastAPI server on localhost. It's a transformer neural network, plus some python code for extra processing. The python code is trivial and could easily be reimplemented in another language. The network needs a lot of compute and each request should be considered expensive.
So far, I've set up authentication with Firebase, to restrict access to the database. But the FastAPI routes (which are stateless and don't reveal any user data) are not authenticated. When making this service public, I need some form of gatekeeping which allows me to limit the number of requests, prevent unauthenticated users, etc.
I'm not sure what the best practices for deployment are. Here's what I've been considering:
Find a provider that hosts the Docker container. Set up user authentication with a FastAPI route, which provides a JWT token. Only allow access to the expensive machine learning API with a valid token. The drawbacks: I think then I'd also have to move the database into the docker container. I don't want to force my users to log in twice, into separate services. And sharing the login credentials or the JWT token across two services doesn't seem feasible. My quick&dirty solution for this would be to hook up sqlite to FastAPI. But 1) I dont' want to worry about data loss. Firebase has convenient features for periodic backup etc. Sure, I can set this up myself, too. But I think proper database management quickly becomes a rabbit hole. And 2) the database traffic will be much much higher than the machine learning API traffic. I suspect that this will increase hosting costs: I'd need a very big virtual machine with a lot of bandwith. If I keep the two components separate, then I need a high-bandwith BAAS database plus a low-bandwith but high-compute ML API.
Set up a two-hop architecture. The machine learning API is not accessible to the client, but only to the cloud functions (or edge functions) of my BAAS provider. There was a good article from supabase https://supabase.com/docs/guides/ai/hugging-face. The supabase article talks specifically about huggingface, but this approach seems universal. Basically: Find a hosting provider for the Docker container and create a secret key. This key is kept in the codebase for the edge functions of the BAAS. An authenticated client can invoke such a function, but it doesn't see the code of the function itself, and therefore can't obtain the key. This means the expensive API calls are always routed through my own code, and I can rate limit or otherwise restrict them as I see fit. The problem here is: I find it hard to compute the pricing. And I worry that my whole software architecture will start to rely on a very large number of edge function invocations. This would make me vulnerable to sudden pricing changes from the provider.
This seems like a rather basic problem, and I've searched the site for related questions. So far I didn't find anything for this specific use-case of a high-bandwith API plus a high-compute API, and potentially using edge functions.
