Questions tagged [big-data]
The big-data tag has no usage guidance.
1
vote
2answers
103 views
Designing a big data web app
How do you design a website that allows users to query a large amount of user data, more specifically:
there are ~100 million users with ~100TB of data, data is stored in HDFS (not a database)
number ...
6
votes
1answer
219 views
How can I reduce the amount of storage needed for a gravitational n-body simulation?
I am currently attempting to create a gravitational n-body simulation using a modified Barnes-Hut algorithm, to be more amicable to GPU computation. This is primarily as a learning project. My goal is ...
1
vote
2answers
231 views
Why is the whole Hadoop ecosystem written in Java?
Developing Big Data processing pipelines and storage, you probably come across software which is more or less a part of the Hadoop ecosystem. Be it Hadoop itself, Spark/Flink, HBase, Kafka, Accumulo, ...
1
vote
1answer
45 views
How to incrementally update value of features in a machine learning pipeline?
I am working on a machine learning pipeline where we have to compute certain measures on streaming data. Every day, new raw data enters our pipeline. To update our features, we have to run an ETL that ...
0
votes
1answer
51 views
How to track change of JSON data over time for large number of entities?
I have a system that checks the status of a large number of entities on schedule every minute. For each entity, there would be a JSON file which has fields indicating the statuses for different ...
1
vote
2answers
831 views
How to process huge amount of data with limited processing ressources? [closed]
I receive daily from an external source a very large amount of data (around 250GB with 260 million rows of fixed width text) distributed over 5 text files. I am writing a Java application that should ...
0
votes
0answers
21 views
Hbase Spout or read all data from Hbase in Apache Storm
I have a Hbase table with around 200 million rows with keys as sha hash, so distributed from 00...00 to ff...ff. I want to either write a spout which gives me all the rows(preferable in batches of n) ...
1
vote
2answers
96 views
How to avoid clashes in distributed computing when polling for work to do?
A big data job is split up into X partitions. The partitions are stored in a database. Status on each partition is also stored in the database and is used to ensure that each partition is only ...
0
votes
1answer
89 views
Application analytics - Logging with NoSQL/RDBMS, BigData [closed]
I need to architect a system for my company. My question is specific to logging. Part of the requirement is to log 2 kinds of data.
A1. User events (Ex: UserA clicked on Button1, performed Action#21. ...
2
votes
2answers
281 views
Is manually tagging content an ideal start for eventual machine learning?
I'm putting together a mobile app that would allow users to search, rate, and upload photos of dishes at restaurants. I have no machine learning experience (just FE/BE/Mobile) so for the time being I ...
0
votes
2answers
106 views
The best approach for processing a large data on php
I am searching for the best approach for process a large amount data using PHP.
I want to create a xml file from a database, when I execute my code the system will stop with a "PHP timeout" error, I ...
0
votes
1answer
341 views
Fast storage format for huge point clouds (fast read/write)
I am working with huge point clouds (1 billion points). I need to process and display them but because of the size of the data, I can't have it all in memory at the same time.
First of all, I am not ...
0
votes
0answers
34 views
Can data in a rooted tree be processed in parallel?
I have data that is structured like the following:
users:
id | name | parent_id
1 | Bob | NULL
2 | Jan | 1
3 | Mat | 2
4 | Irene | 2
5 | Ellie | 2
6 | Laura | 5
7 | Uma | 6
...
0
votes
0answers
97 views
Is MapReduce a correct framework for Extract, Transform, Load of data?
EDIT
I am working on a project to update a legacy ETL infrastructure that supports a number of clients, each with a slightly different setup.
Constraints that cannot be changed:
Source data can ...
2
votes
1answer
190 views
How to make a basic database application, previously developed for 500 records, ready for 1 million records?
I have written a basic web app in PHP, using MongoDB as database engine. The app basically inserts records into database as they become available, and queries the database and displays the data as on ...
0
votes
0answers
112 views
Non HBase solution for huge data that has update and delete in sequential manner
I have to design an application where there are around 5K structured base text files (file.txt) with data and format as below:
Primary key is OrgId + ItemId
OgId|^|ItemId|^|segmentId|^|Sequence|^|...
-1
votes
1answer
350 views
Database design considerations for extremely large data volumes
I have an upcoming interview for a position at a company that deals with multi-petabyte scale data volumes. They're going to grill me about standard database design questions, but what are the best ...
1
vote
1answer
167 views
Moving from Batch to Streaming architecture using a Graph datastructure
The use case I am trying to solve is, assigning millions of users to their groups/segments. I have thousands of different criteria from which buckets of users are created. E.g. bucket criteria:
All ...
0
votes
1answer
751 views
Is this Big Data architecture good enough to handle many requests per second?
I want to ask for a review of my big data app plan. I haven’t much experience in that field, so every single piece of advice would be appreciated.
Here is a link to a diagram of the architecture: My ...
5
votes
4answers
798 views
Can someone explain the technicalities of MapReduce in layman's terms?
When people talk about MapReduce you think about Google and Hadoop. But what is MapReduce itself? How does it work? I came across this blog post that tries to explain just MapReduce without Hadoop, ...
8
votes
3answers
8k views
How to store large amounts of _structured_ data?
The application will continuously (approximately every second) collect the location of users and store them.
This data is structured. In a relational database, it would be stored as:
| user | ...
3
votes
1answer
443 views
Is Machine Learning a part of Data Science?
I am having trouble differentiating between Big Data vs Data Science. From what I've understood so far, both are buzzwords for the old statistics job. Where does machine learning fit into this?
1
vote
1answer
537 views
Near real-time analytics from Cassandra with frequent updates
We have an activity metrics page where users can select a date period and see other user's aggregated activity (by action) and optionally filter everything by 4 or 5 fields. Actions happen ...
10
votes
1answer
1k views
Partial name matching in millions of records
We have developed a web based application for name matching. It operates by breaking names into parts and the Soundex value of each part is stored in a database. The Levenshtein distance metric is ...
0
votes
1answer
104 views
Best Azure Solution for Complex Search Index
I need to perform quick searches against a combination of tags while including date ranges:
Example:
Users
who have requested notifications
who did not respond to a notification sent at least 3 ...
3
votes
1answer
766 views
How is one or more aggregate function implemented in most SQL engines?
In the book Database Fundamentals, Silberschatz. It is explained that aggregate functions can be calculated on the march.
This make sense. What it means is that for calculating the maximun, average ...
1
vote
0answers
32 views
Offsetting high usage in a multi-tenant database environment?
I'm writing a research essay over distributed cloud-based systems, and one of the topics that I'm trying to touch on is potential solutions to high database utilization in a single-database multi-...
-3
votes
1answer
167 views
Should I use NoSQL or HDFS for storage?
I have millions of tweets currently stored in HDFS and I plan to analyze them from Spark (Data mining, text mining, Frequent Term-Based Text Clustering, Social Network Analysis) however, do not know ...
0
votes
1answer
69 views
Does it make sense to store daily snapshots of data and analysis?
I just started working on some data analyzing for an e-commerce site that is moving about 250,000 packages per year. It is very seasonal and need to try to gain insights on both a macro and micro ...
3
votes
0answers
615 views
Best practices for dashboard of near real-time analytics
I’m currently building a dashboard to view some analytics about the data generated by my company's product.
We use MySQL as our database. The SQL queries to generate the analytics from the raw live ...
3
votes
3answers
1k views
Relational vs Graph Database for (initially) moderately-sized network
We're developing an application whose data domain (or at least 90% of it) can be modeled effectively using a relational database. We've been using PostgreSQL since the beginning and have had no ...
0
votes
2answers
3k views
How to properly index MongoDB with ElasticSearch?
we are working on a JAVA EE project which handles huge amount of data, but has to provide full-text-search option (in hungarian language).
So we started to think about what kind of architecture could ...
1
vote
1answer
1k views
Feeding events to Elasticsearch, do I really need Logstash?
I'm designing an analytics system that feeds all events to Elasticsearch. The event lifecycle is as follows:
Visitor does something.
Custom analytics server gathers data, makes an event out of it and ...
2
votes
1answer
982 views
How to architect frequent updates in No-sql db (Cassandra) - architecture
I'm storing some data in Cassandra, then after analyzing it puts in several table, I have its aggregation as daily, weekly, monthly, yearly basis. But after some time if some user reads the content I'...
1
vote
1answer
122 views
BigData: time-based word count
What we are trying to do:
We're trying to build a system that will count the number of unique entries for a certain timeframe. It's working ok until the data grows up or the timeframe increase, then ...
5
votes
1answer
184 views
Querying large amount of data for parallel processing [closed]
I have a dataset containing list of users (around 50M).
Each user has an email address, name, and some more data columns.
I want to send a weekly email to those users, and the content of the email ...
1
vote
0answers
86 views
Dealing with big data [closed]
I am on a project dealing with a lot of data in the form of images and videos (Data related to wind engineering). My requirement is to build a predictive algorithm based on the data I have. I have ...
8
votes
1answer
846 views
quantitatively comparing AST shapes
How could one compare the shape of abstract syntax trees of similar source code programs (C, C++, Go, or anything compiled with GCC...)?
I guess that plagiarism detection on source code would use ...
4
votes
4answers
242 views
Approaches for storing and analysing large amounts of time-based data
I've been asked to develop a "telemetry" application that records data generated by a hardware device, which I read every 100ms.
There are approx 250 data points (32-bit values), but only a subset ...
1
vote
1answer
335 views
Is hadoop designed only for “simple” data processing jobs, where communications between the distributed nodes are sparse?
I am not a professional coder, but rather an engineer/mathematician that uses computer to solve numerical problems. So far most of my problems are math-related, such as solving large scale linear ...
2
votes
1answer
422 views
Random forest ML algorithm suitable for use on cluster based HPC?
I need help in identifying a better algorithm.
I have developed a script using pythons scipy package to analyse a rather large model that I wish to solve. The model contains over 12GB of data ...
0
votes
4answers
26k views
How to insert/update a large amount of data into mysql using php
I have an excel sheet which has a large amount of data. I am using php to insert the data into mysql server.
I have two problems
1) I have to update a row if the id already exists, else insert the ...
6
votes
4answers
2k views
Design of high performance file processing web application
I'm trying to design a web app with ability to scale but can't wrap my heads around few concepts. I want to design it right but im not a experienced programmer, i have more of a system engineering ...
4
votes
3answers
335 views
Big Data: Can it be pre-processed?
My question is about "big data". Basically, big data involves the analysis of a large amount of data to make meaningful insights from it.
I would like to know:
Whether or not large amounts of data ...
8
votes
4answers
9k views
Why Big Data Needs To Be Functional?
I started working on a new project lately related to Big Data for my internship.
My managers recommended to start learning functional programming (They highly recommended Scala).
I had a humbled ...
0
votes
2answers
238 views
Finding patterns in logs
I am trying to put together a way to figure out if incidents occurred based on log content. Typically, on log (or DB table) would contain a list of transactions composed as follow:
{Timestamp} {...
1
vote
1answer
233 views
How to cross-reference many character encodings with ASCII OR UTFx?
I'm working with a binary structure, the goal of which is to index the significance of specific bits for any character encoding so that we may trigger events while doing specific checks against the ...
1
vote
0answers
183 views
What's a good approach for updating millions of rows independently?
I have a MongoDB database with millions of users in collection. An aggregation function returns millions of results with the following format:
Array ( [_id] => Array ( [user] =>
MongoId ...
1
vote
2answers
2k views
Data Aggregation of CSV files java
I have k csv files (5 csv files for example), each file has m fields which produce a key and n values. I need to produce a single csv file with aggregated data.
I'm looking for the most efficient ...
0
votes
1answer
285 views
QFS (Quantcast File System): Scalable for medium “big data” problems or only for extremely large
I am doing some investigating and research into programming "big data", and am particularly interested in QuantCast (QFS) because it is written in my own favourite programming language, C++.
Looking ...
