Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [big-data]

The tag has no usage guidance.

1
vote
1answer
36 views

How to incrementally update value of features in a machine learning pipeline?

I am working on a machine learning pipeline where we have to compute certain measures on streaming data. Every day, new raw data enters our pipeline. To update our features, we have to run an ETL that ...
0
votes
1answer
43 views

How to track change of JSON data over time for large number of entities?

I have a system that checks the status of a large number of entities on schedule every minute. For each entity, there would be a JSON file which has fields indicating the statuses for different ...
1
vote
2answers
397 views

How to process huge amount of data with limited processing ressources? [closed]

I receive daily from an external source a very large amount of data (around 250GB with 260 million rows of fixed width text) distributed over 5 text files. I am writing a Java application that should ...
0
votes
0answers
16 views

Hbase Spout or read all data from Hbase in Apache Storm

I have a Hbase table with around 200 million rows with keys as sha hash, so distributed from 00...00 to ff...ff. I want to either write a spout which gives me all the rows(preferable in batches of n) ...
1
vote
2answers
93 views

How to avoid clashes in distributed computing when polling for work to do?

A big data job is split up into X partitions. The partitions are stored in a database. Status on each partition is also stored in the database and is used to ensure that each partition is only ...
0
votes
1answer
80 views

Application analytics - Logging with NoSQL/RDBMS, BigData [closed]

I need to architect a system for my company. My question is specific to logging. Part of the requirement is to log 2 kinds of data. A1. User events (Ex: UserA clicked on Button1, performed Action#21. ...
2
votes
2answers
198 views

Is manually tagging content an ideal start for eventual machine learning?

I'm putting together a mobile app that would allow users to search, rate, and upload photos of dishes at restaurants. I have no machine learning experience (just FE/BE/Mobile) so for the time being I ...
0
votes
2answers
89 views

The best approach for processing a large data on php

I am searching for the best approach for process a large amount data using PHP. I want to create a xml file from a database, when I execute my code the system will stop with a "PHP timeout" error, I ...
0
votes
1answer
280 views

Fast storage format for huge point clouds (fast read/write)

I am working with huge point clouds (1 billion points). I need to process and display them but because of the size of the data, I can't have it all in memory at the same time. First of all, I am not ...
0
votes
0answers
34 views

Can data in a rooted tree be processed in parallel?

I have data that is structured like the following: users: id | name | parent_id 1 | Bob | NULL 2 | Jan | 1 3 | Mat | 2 4 | Irene | 2 5 | Ellie | 2 6 | Laura | 5 7 | Uma | 6 ...
0
votes
0answers
93 views

Is MapReduce a correct framework for Extract, Transform, Load of data?

EDIT I am working on a project to update a legacy ETL infrastructure that supports a number of clients, each with a slightly different setup. Constraints that cannot be changed: Source data can ...
2
votes
1answer
189 views

How to make a basic database application, previously developed for 500 records, ready for 1 million records?

I have written a basic web app in PHP, using MongoDB as database engine. The app basically inserts records into database as they become available, and queries the database and displays the data as on ...
0
votes
0answers
85 views

What would be good architecture options for multi-index data aggregation?

We have a use case where we need to store a large set (many millions) of data "objects" with multiple attributes (that may differ over time/for different "collections", e.g. for diffrent "type" of ...
0
votes
0answers
111 views

Non HBase solution for huge data that has update and delete in sequential manner

I have to design an application where there are around 5K structured base text files (file.txt) with data and format as below: Primary key is OrgId + ItemId OgId|^|ItemId|^|segmentId|^|Sequence|^|...
-1
votes
1answer
331 views

Database design considerations for extremely large data volumes

I have an upcoming interview for a position at a company that deals with multi-petabyte scale data volumes. They're going to grill me about standard database design questions, but what are the best ...
1
vote
1answer
165 views

Moving from Batch to Streaming architecture using a Graph datastructure

The use case I am trying to solve is, assigning millions of users to their groups/segments. I have thousands of different criteria from which buckets of users are created. E.g. bucket criteria: All ...
0
votes
1answer
702 views

Is this Big Data architecture good enough to handle many requests per second?

I want to ask for a review of my big data app plan. I haven’t much experience in that field, so every single piece of advice would be appreciated. Here is a link to a diagram of the architecture: My ...
5
votes
4answers
708 views

Can someone explain the technicalities of MapReduce in layman's terms?

When people talk about MapReduce you think about Google and Hadoop. But what is MapReduce itself? How does it work? I came across this blog post that tries to explain just MapReduce without Hadoop, ...
8
votes
3answers
7k views

How to store large amounts of _structured_ data?

The application will continuously (approximately every second) collect the location of users and store them. This data is structured. In a relational database, it would be stored as: | user | ...
3
votes
1answer
424 views

Is Machine Learning a part of Data Science?

I am having trouble differentiating between Big Data vs Data Science. From what I've understood so far, both are buzzwords for the old statistics job. Where does machine learning fit into this?
0
votes
1answer
513 views

Near real-time analytics from Cassandra with frequent updates

We have an activity metrics page where users can select a date period and see other user's aggregated activity (by action) and optionally filter everything by 4 or 5 fields. Actions happen ...
0
votes
0answers
74 views

Algorithm to efficiently chart endless signal

A node process of mine receives a sample point every half a second, and I want to update the history chart of all the sample points I receive. The chart should be an array which contains the ...
10
votes
1answer
956 views

Partial name matching in millions of records

We have developed a web based application for name matching. It operates by breaking names into parts and the Soundex value of each part is stored in a database. The Levenshtein distance metric is ...
0
votes
1answer
104 views

Best Azure Solution for Complex Search Index

I need to perform quick searches against a combination of tags while including date ranges: Example: Users who have requested notifications who did not respond to a notification sent at least 3 ...
3
votes
1answer
723 views

How is one or more aggregate function implemented in most SQL engines?

In the book Database Fundamentals, Silberschatz. It is explained that aggregate functions can be calculated on the march. This make sense. What it means is that for calculating the maximun, average ...
1
vote
0answers
31 views

Offsetting high usage in a multi-tenant database environment?

I'm writing a research essay over distributed cloud-based systems, and one of the topics that I'm trying to touch on is potential solutions to high database utilization in a single-database multi-...
-3
votes
1answer
166 views

Should I use NoSQL or HDFS for storage?

I have millions of tweets currently stored in HDFS and I plan to analyze them from Spark (Data mining, text mining, Frequent Term-Based Text Clustering, Social Network Analysis) however, do not know ...
0
votes
1answer
67 views

Does it make sense to store daily snapshots of data and analysis?

I just started working on some data analyzing for an e-commerce site that is moving about 250,000 packages per year. It is very seasonal and need to try to gain insights on both a macro and micro ...
3
votes
0answers
591 views

Best practices for dashboard of near real-time analytics

I’m currently building a dashboard to view some analytics about the data generated by my company's product. We use MySQL as our database. The SQL queries to generate the analytics from the raw live ...
3
votes
3answers
1k views

Relational vs Graph Database for (initially) moderately-sized network

We're developing an application whose data domain (or at least 90% of it) can be modeled effectively using a relational database. We've been using PostgreSQL since the beginning and have had no ...
0
votes
2answers
3k views

How to properly index MongoDB with ElasticSearch?

we are working on a JAVA EE project which handles huge amount of data, but has to provide full-text-search option (in hungarian language). So we started to think about what kind of architecture could ...
1
vote
1answer
1k views

Feeding events to Elasticsearch, do I really need Logstash?

I'm designing an analytics system that feeds all events to Elasticsearch. The event lifecycle is as follows: Visitor does something. Custom analytics server gathers data, makes an event out of it and ...
2
votes
1answer
947 views

How to architect frequent updates in No-sql db (Cassandra) - architecture

I'm storing some data in Cassandra, then after analyzing it puts in several table, I have its aggregation as daily, weekly, monthly, yearly basis. But after some time if some user reads the content I'...
1
vote
1answer
121 views

BigData: time-based word count

What we are trying to do: We're trying to build a system that will count the number of unique entries for a certain timeframe. It's working ok until the data grows up or the timeframe increase, then ...
5
votes
1answer
184 views

Querying large amount of data for parallel processing [closed]

I have a dataset containing list of users (around 50M). Each user has an email address, name, and some more data columns. I want to send a weekly email to those users, and the content of the email ...
1
vote
0answers
85 views

Dealing with big data [closed]

I am on a project dealing with a lot of data in the form of images and videos (Data related to wind engineering). My requirement is to build a predictive algorithm based on the data I have. I have ...
8
votes
1answer
815 views

quantitatively comparing AST shapes

How could one compare the shape of abstract syntax trees of similar source code programs (C, C++, Go, or anything compiled with GCC...)? I guess that plagiarism detection on source code would use ...
4
votes
4answers
241 views

Approaches for storing and analysing large amounts of time-based data

I've been asked to develop a "telemetry" application that records data generated by a hardware device, which I read every 100ms. There are approx 250 data points (32-bit values), but only a subset ...
0
votes
1answer
332 views

Is hadoop designed only for “simple” data processing jobs, where communications between the distributed nodes are sparse?

I am not a professional coder, but rather an engineer/mathematician that uses computer to solve numerical problems. So far most of my problems are math-related, such as solving large scale linear ...
2
votes
1answer
415 views

Random forest ML algorithm suitable for use on cluster based HPC?

I need help in identifying a better algorithm. I have developed a script using pythons scipy package to analyse a rather large model that I wish to solve. The model contains over 12GB of data ...
0
votes
4answers
26k views

How to insert/update a large amount of data into mysql using php

I have an excel sheet which has a large amount of data. I am using php to insert the data into mysql server. I have two problems 1) I have to update a row if the id already exists, else insert the ...
6
votes
4answers
2k views

Design of high performance file processing web application

I'm trying to design a web app with ability to scale but can't wrap my heads around few concepts. I want to design it right but im not a experienced programmer, i have more of a system engineering ...
4
votes
3answers
334 views

Big Data: Can it be pre-processed?

My question is about "big data". Basically, big data involves the analysis of a large amount of data to make meaningful insights from it. I would like to know: Whether or not large amounts of data ...
8
votes
4answers
9k views

Why Big Data Needs To Be Functional?

I started working on a new project lately related to Big Data for my internship. My managers recommended to start learning functional programming (They highly recommended Scala). I had a humbled ...
0
votes
2answers
235 views

Finding patterns in logs

I am trying to put together a way to figure out if incidents occurred based on log content. Typically, on log (or DB table) would contain a list of transactions composed as follow: {Timestamp} {...
1
vote
1answer
232 views

How to cross-reference many character encodings with ASCII OR UTFx?

I'm working with a binary structure, the goal of which is to index the significance of specific bits for any character encoding so that we may trigger events while doing specific checks against the ...
1
vote
0answers
183 views

What's a good approach for updating millions of rows independently?

I have a MongoDB database with millions of users in collection. An aggregation function returns millions of results with the following format: Array ( [_id] => Array ( [user] => MongoId ...
1
vote
2answers
2k views

Data Aggregation of CSV files java

I have k csv files (5 csv files for example), each file has m fields which produce a key and n values. I need to produce a single csv file with aggregated data. I'm looking for the most efficient ...
0
votes
1answer
278 views

QFS (Quantcast File System): Scalable for medium “big data” problems or only for extremely large

I am doing some investigating and research into programming "big data", and am particularly interested in QuantCast (QFS) because it is written in my own favourite programming language, C++. Looking ...
2
votes
1answer
240 views

Experimenting with “big data” and map reduce

I was thinking about experimenting a bit with using mapreduce and such on a dataset we have to see if we can benefit from it in any way. What route should I go if I want to be able to instantly, ...