2

tldr at the bottom if you don't want to read all this! :)

First of all the db I'm using is MongoDB! So I've been building a fun project and all has been well but I hit a small problem. Effectively, the project could be thought of as a simple button on a website. For each click of the button, the click will be sent to the back end via sockets and recorded and stored in a database. This is very small and easy to do as I just have a simple Mongo database with a key called "buttonClicks" that increment by 1 each time it's clicked. There's also some other arbritrary incremental data sent as well like for examples sake, the browser of which the click was sent from. Regardless, the schema currently looks like this

{
    name: Admin
    buttonClicks: 22,
    whichBrowser: {chrome: 10, edge: 12}
}

As you can see, this is all incremental data, and since there's only a handful of browsers on the planet, the whichBrowser object won't get that big.

The problem is I also need a time stamp. What I mean is the way it's setup now, I just know how many times the button was clicked from day 1 till now. What if I want to see how much button clicks there was today/week/month. I plan to show this in a graph so I would also like to bin it by X amount as well. So I may want to graph it where it shows me the clicks through the past day in 5 minute intervals, or clicks through the past week in 10 minute intervals, etc.

How can I achieve this? As it is right now my database is very small since it basically has 1 key with a value that simply increments, but if I were to store the time stamp as is for each click, I imagine the database will get MASSIVE with the "buttonClicks" key having a value of a huge object, something like

{
    name: Admin
    buttonClicks: [{
        date: "3/27/2020 - 12:44:15",
        browser: chrome
    },{
        date: "3/27/2020 - 12:44:15",
        browser: chrome
    },{
        date: "3/27/2020 - 12:44:15",
        browser: edge
    },{
        date: "3/27/2020 - 12:44:16",
        browser: chrome
    },{
        date: "3/27/2020 - 12:44:16",
        browser: fire fox
    }]
}

As you can see, only 5 clicks were recorded but it's already so big. Not to mention (in this example) only 2 seconds has passed and this array already has 5 objects.

Thank you! So does anyone have any advice on how to can store each click (and other info like browser)

tldr: A button is clicked on a site and recorded in the back end. It takes little space for now since it's just incremental data (just up the value by 1 when a button click is recorded) but I realized I need the date and time as well for data visualization purposes. How can I record the data in a way for each click so it won't take a lot of space and I can easily manipulate it (like how much clicks were there the past week, past 10 days, etc)

3

Your main problem here is not the anmount of data (mongo can handle BIG stuff) but the maximum document size of 16 MB.

Thus, you need some kind of "paging" like a log-rotation on a file system. A standard approach is, that you start off with a document per given time interval, e.g. a day:

{
    "name": "Admin",
    "recordingStart": "3/27/2020",
    "buttonClicks": [...]
}

In many cases, this will be enough already.

But maybe your site is high-traffic, so if this grows too large, add a continuation document, i.e.

{
    "name": "Admin",
    "recordingStart": "3/27/2020",
    "ordinal": 1,
    "buttonClicks": [...]
    "continueWith": 2
}

{
    "name": "Admin",
    "recordingStart": "3/27/2020",
    "ordinal": 2,
    "buttonClicks": [...]
}

With a decent base time interval, this can also ease your later aggregations a lot.

6
  • Oh okay cool that makes sense, thanks! So one thing I didn't think was important but forgot to mention was that there could be multiple admins with each having their own site they receive hits/data from. Would that affect the 16mb limit? So each admin will have their own document, then each document will have a sub document which contains what you wrote above. So wouldn't the root document exceed 16mb? The structure is [root document with attribute username, pass, email, website] -> [[sub document of website with name/recording/clicks for day 1], [another sub document for next day], [etc]]
    – WildWombat
    Mar 28 '20 at 16:44
  • No, just use a different document per user.
    – mtj
    Mar 28 '20 at 16:52
  • Right so let's say there's 5 "admins" so 5 websites. Then there will be 5 root documents and inside each will be what you described up there. Even though you split the recordings into the day and make the sub documents not exceed 16mb, wouldn't the root documents (so the 5 root documents) exceed 16mb since I assume it takes into account the size of the sub document? Sorry I'm quite new to databases so I'm trying to wrap my head around structuring them.
    – WildWombat
    Mar 28 '20 at 17:00
  • No, each document individually is limited to 16 MB, there is (practically) no limit to the number of documents in the collection. So you have { day1, admin1, number1 }, { day1, admin1, number2 }, { day1, admin2, number1 }, ... { day2, admin1, ... } ... as individual documents. You should not have these as sub-documents of a single root-document.
    – mtj
    Mar 29 '20 at 5:38
  • Ohh I see, thank you! Would this cause scaling issues? I imagine my collection will become huge very fast but the documents themselves will be quite small.
    – WildWombat
    Mar 29 '20 at 18:41
0

Maybe borrow some textbook "tricks" from compression algorithms?

{
    name: Admin
    buttonClicks: [{
        date: "3/27/2020 - 12:44:15",
        browser: chrome
    },{
        date: "3/27/2020 - 12:44:15",
        browser: chrome
    },{
        date: "3/27/2020 - 12:44:15",
        browser: edge
    },{
        date: "3/27/2020 - 12:44:16",
        browser: chrome
    },{
        date: "3/27/2020 - 12:44:16",
        browser: fire fox
    }]
}

Consider a dictionary:

startDate: "3/27/2020 - 12:44:15"
chrome: 1
edge: 2
firefox: 3

And use the convention that seconds represents seconds passed since startDate.

Your original data then becomes:

{
    name: Admin
    buttonClicks: [{
        seconds: 0
        browser: 1
    },{
        seconds: 0,
        browser: 1
    },{
        seconds: 0,
        browser: 2
    },{
        seconds: 1,
        browser: 1
    },{
        seconds: 1,
        browser: 3
    }]
}

Does this save some space?

-1

I try using long for internal operations and storage, only in View layer they are converted to a human readable format

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.