5

We’re building a platform of different APIs (every API is written by different team, different timezone). We want to implement unified analytics for all of the APIs to have one data lake as a single source of truth. We are using Apigee as our API proxy for all of the APIs. Now the question is: where would you implement the analytics layer?

Our first thought was to implement it in Apigee and log events for every API call - but we encounter problems with providing full request, response payloads since we’re using streaming configurations for performance reasons what prevents Apigee from having access to the payload.

The other approach we considered was to obligate all services to send their own events - but is it really possible? Every team has their own schedule and it seems that analytics would always get lower priority, and how can we really make sure that every API call is being logged as expected?

How do your companies deal with analytics? Where would you implement such a layer? Would be grateful to hear your ideas.

2
  • 2
    If you log the fact that the request happened in Apigee then you can correlate if there are calls which are not logged by the teams. That fact can be shown in monitoring for example. About logging all contents of all requests: That will be quite some data, is that really what you need?
    – Luc Franken
    Commented Aug 9, 2016 at 13:28
  • 1
    A rule of thumb I like from Scott Meyers "Make Interfaces Easy to Use Correctly and Hard to Use Incorrectly". Requiring the services to send their own events certainly makes things easy to use incorrectly if Analytic logging is a requirement. That should answer your question. However, that doesn't mean there's not some sort of 3rd option, perhaps requiring the services to use some sort of common service local API/library.. Then events for logging could be transparent to the services.
    – Dunk
    Commented Aug 10, 2016 at 18:14

2 Answers 2

Reset to default
1

Honestly,your best bet is to just log everything to stdout and stderr, then have your OS handle log rotate and all that. Each service can ship their logs using standard tools like systemd / syslog to your log aggregator. I like logstash for that, but Splunk will work.

If you then need to ship to a third party, your internal log aggregator can control what gets shipped and how often to control cost and sensitivity.

This is by far the simplest and hardest to screw up approach.

0

Now the question is: where would you implement the analytics layer?

In a separate microservice. The code to send event stream to it should be developed cooperatively by all your teams.

The other approach we considered was to obligate all services to send their own events - but is it really possible?

Why not? If you will be getting to much of the data, just use sampling (log only 1/5, 1/10 or 1/100 of all successfull requests + all failed ones).

Every team has their own schedule and it seems that analytics would always get lower priority

I think this is not a technical question, but a question of your project management quality.

and how can we really make sure that every API call is being logged as expected?

I guess the same way you make sure your services operate at all. By tests at all levels and external monitoring.

1
  • I've forgot to mention that there is an intention to let external third party services integrate into our platform in the future. So making the service post the analytics data requires a lot of trust and apart from that, instead of taking care only about their field and how they can add value, they need to also think about implementing analytics calls. So I'm looking for a more generic solution.
    – user2919190
    Commented Aug 10, 2016 at 10:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.