Azure Event Grid API layer beneath a HTTP API layer

Question

I am part of a project/team that is building a new web app in Azure for the first time, having previously built and developed a traditional three tier ASP.NET web app over a number of years.

We have an external architect/consultant helping with the transition to Azure, and they are proposing an architecture that is proving to be somewhat controversial. In very simple terms the server architecture is basically:

   HTTP/REST API (HTTPTriggers) --> Event Grid --> Back-end Microservices (EventTriggers)

I.e., there is an EventGrid abstraction layer between the externally facing API and the back-end 'domain' microservices.

If we take the example of a simple HTTP GET of a data record; the HTTPTrigger C# function sends a 'command' event onto the grid, and waits for an ACKnowledge event, before sending the HTTP response back to the caller.

The abstraction layer isn't super controversial per se, although some have questioned the need for it. There are some benefits I think, such as not having to manage lots of microservice URL endpoints, e.g. if we add a new back-end microservice, or split/merge existing back-end services then the REST API layer can (in principle) be oblivious to these changes at the back-end. There may also be benefits in terms of redundancy and scaling (although one could argue that Azure Functions and use of CosmosDB have those aspects covered without the extra abstraction layer).

The real source of concern is that (as I understand it) the HTTPTrigger function has no way of subscribing to event grid events for its short lifetime (which could/should be sub-second for most/many API calls) in order to receive the ACK event. As such this function sits in a polling loop using 'await Task.Delay()' in each loop so as not to sleep the executing thread, or use excessive CPU. We also talked about backing off the polling frequency over time to get a good balance of low latency for fast ACKs, and minimising the number of the polls/loops for slower ACKs.

The polling loop then, checks some appropriate data store, such as a redis cache entry, or a row in an Azure Table DB. Separately, an Azure Function with an EventTrigger has the sole purpose of handling ACK events and updating that data store. As such, the response data from the back-end microservice is conveyed via that storage, which seems a bit odd for a simple GET request. This use of storage, combined with polling, will add cost and latency, and I think the controversy is largely due to not seeing a clear benefit to counter those issues/costs.

One thing I was wondering about is how this pattern would work if the Azure Functions were geo-distributed (ie., e.g. if a customer wanted distribution of the back-end over two or more data centers); could we configure Azure such that the storage used for the event responses was always local to where the Azure functions were running? I don't think so because there are two Azure Functions - the HTTPTrigger and the EventTrigger, and I'm not sure there is any way of ensuring that the EventTrigger function will run in the same locality as the original HTTPTrigger function - they are two completely independent functions. As such, the state store would need to be geo-distributed in that scenario, which sounds a little crazy to me to have a HTTP response being transmitted via the data replication/synchronisation mechanism of Azure Tables or redis.

Thoughts?

Thanks for reading!

Answer 1

I think this boils down to how self-similar your clients are.

If for example you are the next YouTube, or Twitter. You can expect that any query executed against the backend will resolve to similar/identical response regardless of the user. In such a case, particularly if the cost of computing that response is relatively high, it does make arguable sense to cache the result and drive the entire presentation of the API via that cache. This short-circuits the expensive computations and swaps it for relatively mundane bandwidth/copying issues.

In which case, I would go one further and suggest non-blocking API calls. If they do not see the result in the cache, they return immediately with a user please check again in 5 seconds response. A suitably smart client will make that almost transparent to any real user.

On the other hand If you are more like a Bank, or a the Next Cloud Personal Hard Drive. Then your responses are tailored to each and every user. There is almost no response that would not have to go to the backend, even in the same user session. If there is a cache, its almost universally sensible for this to be in the client, not in your servers.

In this case most of the benefit being bought by the caching is purely operational. It makes deployment easier, and masks the mal-effects of poorly behaving back-end services. If this is the only reason for rely on this, then you have problems. Because you either have bad services(in which case fixing them is the best solution), or you expect to have bad services (in which case you have a culture problem, you really need to fix the malaise producing bad code). In either case this is just a stop-gap, as exactly the same techniques can be applied to the micro-services forming your backend directly, and produce a much better outcome.

The only other benefit is providing a lever for introducing API end-points quickly with specific formatting, etc... I would not argue implementing all the overhead just for this feature, but it can be a surprisingly useful feature for providing end-points for new clients, and backward supporting older clients when the underlying services change.

At the end of the day its a trade off. I would use this only for those parts of the API that share results among many users. Those parts of the api that don't are probably best served by a direct approach.