1

This seems like it should be fairly straight forward but I'm having a devil of a time with it. Hoping some of you fine folks might be able to lend a hand.

I'm using PHP & MySQL, and want to calculate the flow rates of traffic to certain web pages. Ultimately I want to be able to send myself alerts when flow rates rise or fall dramatically over certain time periods. For now I'm thinking I would like to monitor at 5 min, 30min, 60 min, 5 hour, and 24 hour intervals.

The "easy" way to do this would probably be to record every single page request as a separate timestamped entry in my DB and then just count them and do various date diff calculations etc. But I really don't want to create a new row for every request. I would rather just increment counters and work off of those, but I'm completely stumped at the moment.

What would be the easiest way to go about this? My attempts thus far have been so far from the mark that I don't even have any code worth sharing.

2

You're on the right track - each web page request needs to be tracked, but you need to do this is a very simple manner. Then the problem shifts to statistical analysis of these counters, and that's quite easy.

Forget storing each webpage in the DB, instead you need to cache it in memory (as you don't really care about flow rate when the system is restarted) and it'll be much faster to calculate and won't bog down your system writing thousands of counters to the DB.

Don't store the timestamp, instead create a bucket of requests, keep this incremented per minute (which is your minimum granularity), so you have an array of the last 24 hours (1440 entries) with the number of requests each minute stored in each element.

Now you just need to add up the numbers in each of the last n elements, compare them to the current element and if the current number is x over the sum of the others, pop the alert. You could optimise this by performing the sums for the alert periods and storing them in a set of variables every time the minute changes.

  • Interesting... I think this is actually very similar to what I came up with last night, though I'm doing it all in the DB with counters. Can you elaborate on what you mean by caching in memory instead? I'm afraid that's a new one for me. – Jonathan van Clute Jul 20 '15 at 16:41
  • hold it in some application-level state that the web server provides, for PHP that's probably going to be memcached unless you want global variables. You'd best ask another question about that on SO, PHP global state is not really my area of expertise. – gbjbaanb Jul 20 '15 at 18:15
  • OK thanks. I think for my needs I'll stick with DB counters, I don't think it will cause undue load and this is just a proof of concept system anyway. If it basically works well, we'll rebuild it with the real engineering team and "do it right" at that time. Appreciate the suggestions! – Jonathan van Clute Jul 20 '15 at 20:20
  • I just realized my current approach is fundamentally flawed, because it's not using rolling time periods. I don't want to update a 60min counter every 60 mins for example, I want to update it every minute with the last 60 min worth of counted data. I can't for the life of me figure out how to do that without capturing and timestamping a counter every minute and storing thousands of rows of counters in the DB - something I was hoping to avoid. – Jonathan van Clute Jul 20 '15 at 22:40
0

If you want to roll your own, instead of storing based on timestamp, you could use the thread id or equivalent and current minute of day to create a unique key composite key and increment the associated value every time you get a request. Then after each minute, you can aggregate all the requests for that minute across all threads to get a utilization value to use in making your decisions.

Alternatively, there are libraries to do this sort of thing.

  • Hmmm I'm afraid that's a bit over my head, though if there are libraries I could at least have a look. Got any links? – Jonathan van Clute Jul 20 '15 at 2:18
  • There are numerous. I cannot speak to any particular one. hotscripts.com/category/scripts/php/scripts-programs/… Of course there is also google analytics too. – Jason Jul 20 '15 at 2:28
  • yeah GA won't tell me what I need to know, nor will it allow for alerts to be sent in realtime. I know there's a realtime GA API coming but for now I've got to make due. I think I've coded up a simple solution on my own, need to let it roll a few days to see if it's solid. If it is I'll answer the question with what I came up with. – Jonathan van Clute Jul 20 '15 at 6:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.