1

I'm building an API and want to count hits for each user. It's a HTTP API implemented in Python. I could keep the count in a database (using PostGreSQL) but it'll be a very busy API, so I don't want the overhead in the DB. I'm looking at Redis or just writing plain text logs. But is there a better approach?

It's a map tile service, so bursts of hundreds of hits per sec. I want to report usage per month to the user—not keep total count of hits.

6
  • 3
    Read the count from the DB at start. Maintain the count in memory, flushing to the DB periodically and at shutdown.
    – user53141
    Mar 18, 2013 at 16:24
  • 1
    I'm a developer on a system that keeps counts in the database for a similar purpose and I highly advise against this. As your table grows locking will become and issue and it'll have a serious impact on your API's performance. I'd stick the requests in Redis or some other non-relational data store and run a background job that calculates new counts periodically. Are you counting hits for statistical purposes or does each client get a set number of requests? Mar 18, 2013 at 16:41
  • Yes, I want to impose a monthly limit. But also for statistical purposes.
    – mhoegh
    Mar 18, 2013 at 16:53
  • I already running ElasticSearch on my server, so maybe I can use that instead of Redis?
    – mhoegh
    Mar 18, 2013 at 16:56
  • 3
    Might just be me, but if this API and infrastructure is so fragile that a single small write will break the camel's back then you might want to think about re-engineering. Also, look at message busses as that is what you need here. Sometimes they even use hip and cool technologies like redis. Mar 18, 2013 at 18:23

1 Answer 1

1

If you are writing logs anyway, I would advice using a background job to periodically extract the counts from the logs and update the counts in your DB. Consider using a logrotate postrotate script to extract data in order to avoid problems due to rotating logs.
In that scenario any DB is ok, so stick with whatever DB you are already using.

If, however, you do not write logs then I would go with @Shinary's proposal (see comments) to write the hits to redis (or similar) and periodically update from there. As you need this functionality in the context of a web-app, I would advice implementing it as a middleware (and depending on (a) the store you have chosen and (b) the severity of single hits being lost - using asynchronous writes/sends to avoid slowing down the main app.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.