Questions tagged [big-data]
The big-data tag has no usage guidance.
68
questions
1
vote
1answer
32 views
Load for Date dimension table of a warehouse
I have a general question about loading data into a data warehouse (DW).
This is basically a followup to an older question of mine.
I have a general understanding problem about fill the [Date] ...
3
votes
2answers
111 views
Enterprise application warehousing and relational database
I have a general question about design pattern for an enterprise application.
I read a lot about it but its actually hard to find an answer because most you find it rater about how to design a data ...
2
votes
2answers
53 views
Aggregation and storage system design for user event processing?
I have a eCommerce like system which produces 5000 user events (of different kind like product search/product view/profile view) per second
Now for reporting business users would like to view the ...
1
vote
3answers
194 views
Query 30 million HTML documents
I have 30-ish million html documents in a file system. There is no emergency, the files are in a reasonable directory tree, it's not breaking the file system. But I'd like to be able to organize and ...
0
votes
0answers
14 views
Design question: Parquet with “corrections”
We are loading in insurance claims data into Parquet on a daily basis for analysis in Spark. At a high level, each claim record has a date, claimant details and value of the claim. This works well in ...
0
votes
0answers
69 views
Generating fake number for a 25 digit PII number in a file containing millions of rows
I have to expose some sensitive data containing a PII column that has a 25 digit number. Rest of the columns aren't PII data. This is done such that the data can be safely shared to the larger ...
2
votes
0answers
20 views
How to design a report processing model using Spark in the most efficient way
I have a reporting system which gets time-series data from numerous meters (here I am referring it as raw_data)
I need to generate several reports based on different combinations of the incoming ...
1
vote
2answers
137 views
Designing a big data web app
How do you design a website that allows users to query a large amount of user data, more specifically:
there are ~100 million users with ~100TB of data, data is stored in HDFS (not a database)
number ...
6
votes
1answer
238 views
How can I reduce the amount of storage needed for a gravitational n-body simulation?
I am currently attempting to create a gravitational n-body simulation using a modified Barnes-Hut algorithm, to be more amicable to GPU computation. This is primarily as a learning project. My goal is ...
1
vote
3answers
494 views
Why is the whole Hadoop ecosystem written in Java?
Developing Big Data processing pipelines and storage, you probably come across software which is more or less a part of the Hadoop ecosystem. Be it Hadoop itself, Spark/Flink, HBase, Kafka, Accumulo, ...
0
votes
1answer
87 views
How to incrementally update value of features in a machine learning pipeline?
I am working on a machine learning pipeline where we have to compute certain measures on streaming data. Every day, new raw data enters our pipeline. To update our features, we have to run an ETL that ...
1
vote
1answer
197 views
How to track change of JSON data over time for large number of entities?
I have a system that checks the status of a large number of entities on schedule every minute. For each entity, there would be a JSON file which has fields indicating the statuses for different ...
1
vote
2answers
3k views
How to process huge amount of data with limited processing ressources? [closed]
I receive daily from an external source a very large amount of data (around 250GB with 260 million rows of fixed width text) distributed over 5 text files. I am writing a Java application that should ...
1
vote
2answers
126 views
How to avoid clashes in distributed computing when polling for work to do?
A big data job is split up into X partitions. The partitions are stored in a database. Status on each partition is also stored in the database and is used to ensure that each partition is only ...
0
votes
1answer
283 views
Application analytics - Logging with NoSQL/RDBMS, BigData [closed]
I need to architect a system for my company. My question is specific to logging. Part of the requirement is to log 2 kinds of data.
A1. User events (Ex: UserA clicked on Button1, performed Action#21. ...
3
votes
2answers
487 views
Is manually tagging content an ideal start for eventual machine learning?
I'm putting together a mobile app that would allow users to search, rate, and upload photos of dishes at restaurants. I have no machine learning experience (just FE/BE/Mobile) so for the time being I ...
0
votes
2answers
381 views
The best approach for processing a large data on php
I am searching for the best approach for process a large amount data using PHP.
I want to create a xml file from a database, when I execute my code the system will stop with a "PHP timeout" error, I ...
0
votes
1answer
589 views
Fast storage format for huge point clouds (fast read/write)
I am working with huge point clouds (1 billion points). I need to process and display them but because of the size of the data, I can't have it all in memory at the same time.
First of all, I am not ...
0
votes
0answers
36 views
Can data in a rooted tree be processed in parallel?
I have data that is structured like the following:
users:
id | name | parent_id
1 | Bob | NULL
2 | Jan | 1
3 | Mat | 2
4 | Irene | 2
5 | Ellie | 2
6 | Laura | 5
7 | Uma | 6
...
0
votes
0answers
179 views
Is MapReduce a correct framework for Extract, Transform, Load of data?
EDIT
I am working on a project to update a legacy ETL infrastructure that supports a number of clients, each with a slightly different setup.
Constraints that cannot be changed:
Source data can ...
2
votes
1answer
192 views
How to make a basic database application, previously developed for 500 records, ready for 1 million records?
I have written a basic web app in PHP, using MongoDB as database engine. The app basically inserts records into database as they become available, and queries the database and displays the data as on ...
-1
votes
1answer
392 views
Database design considerations for extremely large data volumes
I have an upcoming interview for a position at a company that deals with multi-petabyte scale data volumes. They're going to grill me about standard database design questions, but what are the best ...
1
vote
1answer
182 views
Moving from Batch to Streaming architecture using a Graph datastructure
The use case I am trying to solve is, assigning millions of users to their groups/segments. I have thousands of different criteria from which buckets of users are created. E.g. bucket criteria:
All ...
1
vote
1answer
831 views
Is this Big Data architecture good enough to handle many requests per second?
I want to ask for a review of my big data app plan. I haven’t much experience in that field, so every single piece of advice would be appreciated.
Here is a link to a diagram of the architecture: My ...
5
votes
4answers
1k views
Can someone explain the technicalities of MapReduce in layman's terms?
When people talk about MapReduce you think about Google and Hadoop. But what is MapReduce itself? How does it work? I came across this blog post that tries to explain just MapReduce without Hadoop, ...
9
votes
3answers
11k views
How to store large amounts of _structured_ data?
The application will continuously (approximately every second) collect the location of users and store them.
This data is structured. In a relational database, it would be stored as:
| user | ...
3
votes
1answer
487 views
Is Machine Learning a part of Data Science?
I am having trouble differentiating between Big Data vs Data Science. From what I've understood so far, both are buzzwords for the old statistics job. Where does machine learning fit into this?
1
vote
1answer
593 views
Near real-time analytics from Cassandra with frequent updates
We have an activity metrics page where users can select a date period and see other user's aggregated activity (by action) and optionally filter everything by 4 or 5 fields. Actions happen ...
10
votes
1answer
1k views
Partial name matching in millions of records
We have developed a web based application for name matching. It operates by breaking names into parts and the Soundex value of each part is stored in a database. The Levenshtein distance metric is ...
0
votes
1answer
124 views
Best Azure Solution for Complex Search Index
I need to perform quick searches against a combination of tags while including date ranges:
Example:
Users
who have requested notifications
who did not respond to a notification sent at least 3 ...
3
votes
1answer
1k views
How is one or more aggregate function implemented in most SQL engines?
In the book Database Fundamentals, Silberschatz. It is explained that aggregate functions can be calculated on the march.
This make sense. What it means is that for calculating the maximun, average ...
1
vote
0answers
32 views
Offsetting high usage in a multi-tenant database environment?
I'm writing a research essay over distributed cloud-based systems, and one of the topics that I'm trying to touch on is potential solutions to high database utilization in a single-database multi-...
-3
votes
1answer
171 views
Should I use NoSQL or HDFS for storage?
I have millions of tweets currently stored in HDFS and I plan to analyze them from Spark (Data mining, text mining, Frequent Term-Based Text Clustering, Social Network Analysis) however, do not know ...
0
votes
1answer
72 views
Does it make sense to store daily snapshots of data and analysis?
I just started working on some data analyzing for an e-commerce site that is moving about 250,000 packages per year. It is very seasonal and need to try to gain insights on both a macro and micro ...
3
votes
1answer
767 views
Best practices for dashboard of near real-time analytics
I’m currently building a dashboard to view some analytics about the data generated by my company's product.
We use MySQL as our database. The SQL queries to generate the analytics from the raw live ...
4
votes
3answers
1k views
Relational vs Graph Database for (initially) moderately-sized network
We're developing an application whose data domain (or at least 90% of it) can be modeled effectively using a relational database. We've been using PostgreSQL since the beginning and have had no ...
0
votes
2answers
3k views
How to properly index MongoDB with ElasticSearch?
we are working on a JAVA EE project which handles huge amount of data, but has to provide full-text-search option (in hungarian language).
So we started to think about what kind of architecture could ...
1
vote
1answer
1k views
Feeding events to Elasticsearch, do I really need Logstash?
I'm designing an analytics system that feeds all events to Elasticsearch. The event lifecycle is as follows:
Visitor does something.
Custom analytics server gathers data, makes an event out of it and ...
2
votes
1answer
1k views
How to architect frequent updates in No-sql db (Cassandra) - architecture
I'm storing some data in Cassandra, then after analyzing it puts in several table, I have its aggregation as daily, weekly, monthly, yearly basis. But after some time if some user reads the content I'...
1
vote
1answer
133 views
BigData: time-based word count
What we are trying to do:
We're trying to build a system that will count the number of unique entries for a certain timeframe. It's working ok until the data grows up or the timeframe increase, then ...
5
votes
1answer
189 views
Querying large amount of data for parallel processing [closed]
I have a dataset containing list of users (around 50M).
Each user has an email address, name, and some more data columns.
I want to send a weekly email to those users, and the content of the email ...
1
vote
0answers
90 views
Dealing with big data [closed]
I am on a project dealing with a lot of data in the form of images and videos (Data related to wind engineering). My requirement is to build a predictive algorithm based on the data I have. I have ...
8
votes
1answer
1k views
quantitatively comparing AST shapes
How could one compare the shape of abstract syntax trees of similar source code programs (C, C++, Go, or anything compiled with GCC...)?
I guess that plagiarism detection on source code would use ...
4
votes
4answers
247 views
Approaches for storing and analysing large amounts of time-based data
I've been asked to develop a "telemetry" application that records data generated by a hardware device, which I read every 100ms.
There are approx 250 data points (32-bit values), but only a subset ...
1
vote
1answer
338 views
Is hadoop designed only for “simple” data processing jobs, where communications between the distributed nodes are sparse?
I am not a professional coder, but rather an engineer/mathematician that uses computer to solve numerical problems. So far most of my problems are math-related, such as solving large scale linear ...
2
votes
1answer
452 views
Random forest ML algorithm suitable for use on cluster based HPC?
I need help in identifying a better algorithm.
I have developed a script using pythons scipy package to analyse a rather large model that I wish to solve. The model contains over 12GB of data ...
0
votes
4answers
28k views
How to insert/update a large amount of data into mysql using php
I have an excel sheet which has a large amount of data. I am using php to insert the data into mysql server.
I have two problems
1) I have to update a row if the id already exists, else insert the ...
6
votes
4answers
2k views
Design of high performance file processing web application
I'm trying to design a web app with ability to scale but can't wrap my heads around few concepts. I want to design it right but im not a experienced programmer, i have more of a system engineering ...
4
votes
3answers
338 views
Big Data: Can it be pre-processed?
My question is about "big data". Basically, big data involves the analysis of a large amount of data to make meaningful insights from it.
I would like to know:
Whether or not large amounts of data ...
9
votes
4answers
10k views
Why Big Data Needs To Be Functional?
I started working on a new project lately related to Big Data for my internship.
My managers recommended to start learning functional programming (They highly recommended Scala).
I had a humbled ...
