0

I'm thinking about this in terms of a No SQL database, more specifically MongoDB. So, I want to build something like Google Analytics where I will be taking in a ton of data and when it occurs so I can show how many times X happened this year/month/week/day/hr/etc. Like Google Analytics, other than the time stamp, the only other data being stored is small data. There will also be multiple points from which I'm taking in data, like how there's different admins in Google Analytics with their own dashboard. I'll give an example.

There's 3 "admins" with their own dashboard and website, then we need to take in data from each site. For simplicity sake the only data we'll be taking in is when a user leaves a comment on their page and if the comment had a picture.

This is how I originally had it, but of course there's no time stamp

{
    Name: "Admin 1",
    numComments: 10,
    hadPics: {Yes: 6, No: 4}
},{
    Name: "Admin 2",
    numComments: 12,
    hadPics: {Yes: 6, No: 6}
},{
    Name: "Admin 3",
    numComments: 5,
    hadPics: {Yes: 3, No: 2}
}

Then I changed it to this

{
    Name: "Admin 1",
    numComments: 3,
    time: [{
        time: "3/28/2020 - 6:45:36 am",
        hadPics: "No"
    },{
        time: "3/28/2020 - 6:45:37 am",
        hadPics: "Yes"
    },{
        time: "3/28/2020 - 6:46:10 am",
        hadPics: "Yes"
    }]
},{
    Name: "Admin 2",
    numComments: 5,
    time: [{same structure as above}]
},{
    Name: "Admin 3",
    numComments: 6,
    time: [{same structure as above}]
}

But I imagine this would get insanely huge and is not a good way to store the data, not to mention I also just found out that collections have a max space of 16mb which sounds like something I would pass if I were to store it like that? Like the time array would get very big. This is only from 3 comments, imagine if there were 10 comments each second and there were 100 "Admins". Within a minute the time array would have 100 items in it and I'd have done 1000 writes into the database.

If anyone has any suggestions on how to save this data well so that I can easily retrieve and store the data, I would love to hear it! Especially if it makes it easier when grabbing data like from the last day, week, etc.

Improve this question
1
  • You misunderstood this: the document has a max size of 16 MB, the collection can be as large as you like. Right now, the distinction between a collection and a document seems not to be exactly clear. See also my last comment in your other question.
    – mtj
    Mar 29, 2020 at 5:49

1 Answer 1

Reset to default
1

There are really sophisticated solutions for storing such data. You should take a look at "Time-Series Databases" like InfluxDB, Druid or Prometheus. They come with beautiful administration interfaces, they abstract the most common tasks out-of-the-box and are highly optimized for time-series data.

From a personal point-of-view i would not recommend using NoSQL databases for tasks like this because you are dealing with structured data. NoSQL databases (especially MongoDB) flourish when using Geo-data or other unstructured information like fast changing product sortiments etc.

https://www.influxdata.com/

Improve this answer
5
  • I've seen a few of these time series databases around but I've never really looked into them. Do you have any suggestions on a db that's good for beginner/first time user?
    – WildWombat
    Mar 29, 2020 at 18:40
  • Personally I would start with InfluxDB or Druid. InfluxDB (v1.7) is very quick in setup and uses, just like Druid, some sort of SQL dialect to query data. Though InfluxDB is more commonly used i would prefer this one. But, as i said, this is just from a purely personal point of view. For a productive system i would first look for the pros and cons of each system.
    – null1
    Mar 29, 2020 at 19:11
  • Gotcha, I'll def take a look at those two and weigh my options. Another question I had was are these db's one's that I'll have to host myself? I was planning to go with Mongo and just use Mongos own cloud service "Atlas" since all I have to do is connect to the database and that's that. I imagine setting up/running one of those db's will take a lot more effort?
    – WildWombat
    Mar 29, 2020 at 19:35
  • When we setup databases we basically always go for our own hosting options due to privacy reasons. But AWS is typically a good starting point for cloud hosting if you are interested. Hosting databases can be quite difficult because of scalability of your service (horizontal vs vertical scaling). But for a relatively small system it should be sufficient if you install one locally on your host machine and register it for autostart (systemctl) then you should be able to access it in your application.
    – null1
    Mar 29, 2020 at 20:50
  • I would also recommend the answer to include other major players such as the ELK stack, etc.
    – Rakshith Ravi
    Apr 4, 2020 at 11:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.