Questions tagged [big-data]
The big-data tag has no usage guidance.
75
questions
0
votes
0
answers
49
views
Cloud solution to parse and process 1M+ rows
We are writing a service that can parse a CSV file containing over 1M+ rows. Each row needs insertion (or update) data of a DB record. We are currently using DynamoDB.
What AWS services are best ...
- 117
1
vote
1
answer
183
views
How to store a huge volume of time-series datapoints in an efficient way?
We have an application producing 5k-10k datapoints per second. Each datapoint has more than one metric, alongside its time of creation.
We are looking for an efficient, scalable way to store this huge ...
- 127
0
votes
0
answers
29
views
Django Model Design
I am in the design phase of a project and I am having some trouble finding a solution that can perform queries efficiently with PostgreSQL backend
Say, there exists an item and each item has fields ...
4
votes
1
answer
243
views
How do you perform accumulation on large data sets and pass the results as a response to REST API?
I have around 125 million event records on s3. The s3 bucket structure is: year/month/day/hour/*. Inside each hour directory, we have files for every minute. A typical filename looks like this: ...
- 51
1
vote
0
answers
84
views
How to (simply) architecture a way to ingest multiple types of large files, process them, and send data in chunks to web services?
Note: All of this would be in AWS
Hi everyone,
What would you guys suggest for building something that:
Takes in several different input file types (ex: csv, json, jsonl,
xml, .gz, ...)
That can be ...
user384015
0
votes
0
answers
22
views
Should aggregated data include meta data?
I want to create a aggregation job that executes a big db query and flush it into BigQuery.
My question is should I include only the id of the entities (campaign id, advertiser id, user id) or should ...
- 109
0
votes
1
answer
75
views
A program design question: Good idea using HDFS in c for reading large data?
I have mainly three groups of CSV files (each file is divided into several small files): First group of CSV files have 600+ GB in total (MAYBE 200+ GB if in int, cause CSV calculates by char right?), ...
- 101
2
votes
2
answers
2k
views
From Oracle to Apache Parquet : how to handle eventual consistency?
I have an existing production Oracle Database. However, there are performance issues for certain kind of operations, because of the volume of the data, or the complexity of queries.
That's why I ...
- 31
1
vote
1
answer
353
views
Load for Date dimension table of a warehouse
I have a general question about loading data into a data warehouse (DW).
This is basically a followup to an older question of mine.
I have a general understanding problem about fill the [Date] ...
- 159
3
votes
2
answers
135
views
Enterprise application warehousing and relational database
I have a general question about design pattern for an enterprise application.
I read a lot about it but its actually hard to find an answer because most you find it rater about how to design a data ...
- 159
2
votes
2
answers
672
views
Aggregation and storage system design for user event processing?
I have a eCommerce like system which produces 5000 user events (of different kind like product search/product view/profile view) per second
Now for reporting business users would like to view the ...
- 257
1
vote
3
answers
240
views
Query 30 million HTML documents
I have 30-ish million html documents in a file system. There is no emergency, the files are in a reasonable directory tree, it's not breaking the file system. But I'd like to be able to organize and ...
- 2,797
0
votes
0
answers
77
views
Generating fake number for a 25 digit PII number in a file containing millions of rows
I have to expose some sensitive data containing a PII column that has a 25 digit number. Rest of the columns aren't PII data. This is done such that the data can be safely shared to the larger ...
- 101
2
votes
0
answers
23
views
How to design a report processing model using Spark in the most efficient way
I have a reporting system which gets time-series data from numerous meters (here I am referring it as raw_data)
I need to generate several reports based on different combinations of the incoming ...
1
vote
2
answers
656
views
Designing a big data web app
How do you design a website that allows users to query a large amount of user data, more specifically:
there are ~100 million users with ~100TB of data, data is stored in HDFS (not a database)
number ...
- 131
6
votes
1
answer
252
views
How can I reduce the amount of storage needed for a gravitational n-body simulation?
I am currently attempting to create a gravitational n-body simulation using a modified Barnes-Hut algorithm, to be more amicable to GPU computation. This is primarily as a learning project. My goal is ...
- 879
3
votes
4
answers
1k
views
Why is the whole Hadoop ecosystem written in Java?
Developing Big Data processing pipelines and storage, you probably come across software which is more or less a part of the Hadoop ecosystem. Be it Hadoop itself, Spark/Flink, HBase, Kafka, Accumulo, ...
- 237
-1
votes
1
answer
120
views
How to incrementally update value of features in a machine learning pipeline?
I am working on a machine learning pipeline where we have to compute certain measures on streaming data. Every day, new raw data enters our pipeline. To update our features, we have to run an ETL that ...
3
votes
2
answers
2k
views
How to track change of JSON data over time for large number of entities?
I have a system that checks the status of a large number of entities on schedule every minute. For each entity, there would be a JSON file which has fields indicating the statuses for different ...
- 235
1
vote
2
answers
5k
views
How to process huge amount of data with limited processing ressources? [closed]
I receive daily from an external source a very large amount of data (around 250GB with 260 million rows of fixed width text) distributed over 5 text files. I am writing a Java application that should ...
- 49
1
vote
2
answers
319
views
How to avoid clashes in distributed computing when polling for work to do?
A big data job is split up into X partitions. The partitions are stored in a database. Status on each partition is also stored in the database and is used to ensure that each partition is only ...
- 61
0
votes
1
answer
451
views
Application analytics - Logging with NoSQL/RDBMS, BigData [closed]
I need to architect a system for my company. My question is specific to logging. Part of the requirement is to log 2 kinds of data.
A1. User events (Ex: UserA clicked on Button1, performed Action#21. ...
- 101
3
votes
2
answers
541
views
Is manually tagging content an ideal start for eventual machine learning?
I'm putting together a mobile app that would allow users to search, rate, and upload photos of dishes at restaurants. I have no machine learning experience (just FE/BE/Mobile) so for the time being I ...
- 273
0
votes
2
answers
1k
views
The best approach for processing a large data on php
I am searching for the best approach for process a large amount data using PHP.
I want to create a xml file from a database, when I execute my code the system will stop with a "PHP timeout" error, I ...
- 103
0
votes
1
answer
1k
views
Fast storage format for huge point clouds (fast read/write)
I am working with huge point clouds (1 billion points). I need to process and display them but because of the size of the data, I can't have it all in memory at the same time.
First of all, I am not ...
- 101
0
votes
0
answers
38
views
Can data in a rooted tree be processed in parallel?
I have data that is structured like the following:
users:
id | name | parent_id
1 | Bob | NULL
2 | Jan | 1
3 | Mat | 2
4 | Irene | 2
5 | Ellie | 2
6 | Laura | 5
7 | Uma | 6
...
- 1,386
0
votes
0
answers
664
views
Is MapReduce a correct framework for Extract, Transform, Load of data?
EDIT
I am working on a project to update a legacy ETL infrastructure that supports a number of clients, each with a slightly different setup.
Constraints that cannot be changed:
Source data can ...
- 1,386
2
votes
1
answer
195
views
How to make a basic database application, previously developed for 500 records, ready for 1 million records?
I have written a basic web app in PHP, using MongoDB as database engine. The app basically inserts records into database as they become available, and queries the database and displays the data as on ...
- 131
-1
votes
1
answer
411
views
Database design considerations for extremely large data volumes
I have an upcoming interview for a position at a company that deals with multi-petabyte scale data volumes. They're going to grill me about standard database design questions, but what are the best ...
1
vote
1
answer
195
views
Moving from Batch to Streaming architecture using a Graph datastructure
The use case I am trying to solve is, assigning millions of users to their groups/segments. I have thousands of different criteria from which buckets of users are created. E.g. bucket criteria:
All ...
- 111
1
vote
1
answer
883
views
Is this Big Data architecture good enough to handle many requests per second?
I want to ask for a review of my big data app plan. I haven’t much experience in that field, so every single piece of advice would be appreciated.
Here is a link to a diagram of the architecture: My ...
- 137
7
votes
4
answers
2k
views
Can someone explain the technicalities of MapReduce in layman's terms?
When people talk about MapReduce you think about Google and Hadoop. But what is MapReduce itself? How does it work? I came across this blog post that tries to explain just MapReduce without Hadoop, ...
- 81
11
votes
3
answers
15k
views
How to store large amounts of _structured_ data?
The application will continuously (approximately every second) collect the location of users and store them.
This data is structured. In a relational database, it would be stored as:
| user | ...
- 1,912
3
votes
2
answers
571
views
Is Machine Learning a part of Data Science?
I am having trouble differentiating between Big Data vs Data Science. From what I've understood so far, both are buzzwords for the old statistics job. Where does machine learning fit into this?
- 143
2
votes
1
answer
677
views
Near real-time analytics from Cassandra with frequent updates
We have an activity metrics page where users can select a date period and see other user's aggregated activity (by action) and optionally filter everything by 4 or 5 fields. Actions happen ...
- 129
10
votes
1
answer
2k
views
Partial name matching in millions of records
We have developed a web based application for name matching. It operates by breaking names into parts and the Soundex value of each part is stored in a database. The Levenshtein distance metric is ...
- 229
0
votes
1
answer
194
views
Best Azure Solution for Complex Search Index
I need to perform quick searches against a combination of tags while including date ranges:
Example:
Users
who have requested notifications
who did not respond to a notification sent at least 3 ...
- 101
3
votes
1
answer
1k
views
How is one or more aggregate function implemented in most SQL engines?
In the book Database Fundamentals, Silberschatz. It is explained that aggregate functions can be calculated on the march.
This make sense. What it means is that for calculating the maximun, average ...
- 336
1
vote
0
answers
32
views
Offsetting high usage in a multi-tenant database environment?
I'm writing a research essay over distributed cloud-based systems, and one of the topics that I'm trying to touch on is potential solutions to high database utilization in a single-database multi-...
- 1,317
-3
votes
1
answer
178
views
Should I use NoSQL or HDFS for storage?
I have millions of tweets currently stored in HDFS and I plan to analyze them from Spark (Data mining, text mining, Frequent Term-Based Text Clustering, Social Network Analysis) however, do not know ...
- 9
0
votes
1
answer
140
views
Does it make sense to store daily snapshots of data and analysis?
I just started working on some data analyzing for an e-commerce site that is moving about 250,000 packages per year. It is very seasonal and need to try to gain insights on both a macro and micro ...
- 111
4
votes
1
answer
1k
views
Best practices for dashboard of near real-time analytics
I’m currently building a dashboard to view some analytics about the data generated by my company's product.
We use MySQL as our database. The SQL queries to generate the analytics from the raw live ...
- 141
4
votes
3
answers
2k
views
Relational vs Graph Database for (initially) moderately-sized network
We're developing an application whose data domain (or at least 90% of it) can be modeled effectively using a relational database. We've been using PostgreSQL since the beginning and have had no ...
- 437
0
votes
2
answers
4k
views
How to properly index MongoDB with ElasticSearch?
we are working on a JAVA EE project which handles huge amount of data, but has to provide full-text-search option (in hungarian language).
So we started to think about what kind of architecture could ...
- 111
2
votes
1
answer
2k
views
Feeding events to Elasticsearch, do I really need Logstash?
I'm designing an analytics system that feeds all events to Elasticsearch. The event lifecycle is as follows:
Visitor does something.
Custom analytics server gathers data, makes an event out of it and ...
- 416
2
votes
1
answer
1k
views
How to architect frequent updates in No-sql db (Cassandra) - architecture
I'm storing some data in Cassandra, then after analyzing it puts in several table, I have its aggregation as daily, weekly, monthly, yearly basis. But after some time if some user reads the content I'...
- 121
1
vote
1
answer
144
views
BigData: time-based word count
What we are trying to do:
We're trying to build a system that will count the number of unique entries for a certain timeframe. It's working ok until the data grows up or the timeframe increase, then ...
- 197
5
votes
1
answer
196
views
Querying large amount of data for parallel processing [closed]
I have a dataset containing list of users (around 50M).
Each user has an email address, name, and some more data columns.
I want to send a weekly email to those users, and the content of the email ...
- 159
1
vote
0
answers
94
views
Dealing with big data [closed]
I am on a project dealing with a lot of data in the form of images and videos (Data related to wind engineering). My requirement is to build a predictive algorithm based on the data I have. I have ...
- 216
8
votes
1
answer
1k
views
quantitatively comparing AST shapes
How could one compare the shape of abstract syntax trees of similar source code programs (C, C++, Go, or anything compiled with GCC...)?
I guess that plagiarism detection on source code would use ...
