Questions tagged [big-data]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
49 views

Cloud solution to parse and process 1M+ rows

We are writing a service that can parse a CSV file containing over 1M+ rows. Each row needs insertion (or update) data of a DB record. We are currently using DynamoDB. What AWS services are best ...
user avatar
1 vote
1 answer
183 views

How to store a huge volume of time-series datapoints in an efficient way?

We have an application producing 5k-10k datapoints per second. Each datapoint has more than one metric, alongside its time of creation. We are looking for an efficient, scalable way to store this huge ...
user avatar
  • 127
0 votes
0 answers
29 views

Django Model Design

I am in the design phase of a project and I am having some trouble finding a solution that can perform queries efficiently with PostgreSQL backend Say, there exists an item and each item has fields ...
user avatar
4 votes
1 answer
243 views

How do you perform accumulation on large data sets and pass the results as a response to REST API?

I have around 125 million event records on s3. The s3 bucket structure is: year/month/day/hour/*. Inside each hour directory, we have files for every minute. A typical filename looks like this: ...
user avatar
  • 51
1 vote
0 answers
84 views

How to (simply) architecture a way to ingest multiple types of large files, process them, and send data in chunks to web services?

Note: All of this would be in AWS Hi everyone, What would you guys suggest for building something that: Takes in several different input file types (ex: csv, json, jsonl, xml, .gz, ...) That can be ...
user avatar
0 votes
0 answers
22 views

Should aggregated data include meta data?

I want to create a aggregation job that executes a big db query and flush it into BigQuery. My question is should I include only the id of the entities (campaign id, advertiser id, user id) or should ...
user avatar
  • 109
0 votes
1 answer
75 views

A program design question: Good idea using HDFS in c for reading large data?

I have mainly three groups of CSV files (each file is divided into several small files): First group of CSV files have 600+ GB in total (MAYBE 200+ GB if in int, cause CSV calculates by char right?), ...
user avatar
  • 101
2 votes
2 answers
2k views

From Oracle to Apache Parquet : how to handle eventual consistency?

I have an existing production Oracle Database. However, there are performance issues for certain kind of operations, because of the volume of the data, or the complexity of queries. That's why I ...
user avatar
  • 31
1 vote
1 answer
353 views

Load for Date dimension table of a warehouse

I have a general question about loading data into a data warehouse (DW). This is basically a followup to an older question of mine. I have a general understanding problem about fill the [Date] ...
user avatar
3 votes
2 answers
135 views

Enterprise application warehousing and relational database

I have a general question about design pattern for an enterprise application. I read a lot about it but its actually hard to find an answer because most you find it rater about how to design a data ...
user avatar
2 votes
2 answers
672 views

Aggregation and storage system design for user event processing?

I have a eCommerce like system which produces 5000 user events (of different kind like product search/product view/profile view) per second Now for reporting business users would like to view the ...
user avatar
  • 257
1 vote
3 answers
240 views

Query 30 million HTML documents

I have 30-ish million html documents in a file system. There is no emergency, the files are in a reasonable directory tree, it's not breaking the file system. But I'd like to be able to organize and ...
user avatar
  • 2,797
0 votes
0 answers
77 views

Generating fake number for a 25 digit PII number in a file containing millions of rows

I have to expose some sensitive data containing a PII column that has a 25 digit number. Rest of the columns aren't PII data. This is done such that the data can be safely shared to the larger ...
user avatar
2 votes
0 answers
23 views

How to design a report processing model using Spark in the most efficient way

I have a reporting system which gets time-series data from numerous meters (here I am referring it as raw_data) I need to generate several reports based on different combinations of the incoming ...
user avatar
1 vote
2 answers
656 views

Designing a big data web app

How do you design a website that allows users to query a large amount of user data, more specifically: there are ~100 million users with ~100TB of data, data is stored in HDFS (not a database) number ...
user avatar
  • 131
6 votes
1 answer
252 views

How can I reduce the amount of storage needed for a gravitational n-body simulation?

I am currently attempting to create a gravitational n-body simulation using a modified Barnes-Hut algorithm, to be more amicable to GPU computation. This is primarily as a learning project. My goal is ...
user avatar
  • 879
3 votes
4 answers
1k views

Why is the whole Hadoop ecosystem written in Java?

Developing Big Data processing pipelines and storage, you probably come across software which is more or less a part of the Hadoop ecosystem. Be it Hadoop itself, Spark/Flink, HBase, Kafka, Accumulo, ...
user avatar
  • 237
-1 votes
1 answer
120 views

How to incrementally update value of features in a machine learning pipeline?

I am working on a machine learning pipeline where we have to compute certain measures on streaming data. Every day, new raw data enters our pipeline. To update our features, we have to run an ETL that ...
user avatar
3 votes
2 answers
2k views

How to track change of JSON data over time for large number of entities?

I have a system that checks the status of a large number of entities on schedule every minute. For each entity, there would be a JSON file which has fields indicating the statuses for different ...
user avatar
1 vote
2 answers
5k views

How to process huge amount of data with limited processing ressources? [closed]

I receive daily from an external source a very large amount of data (around 250GB with 260 million rows of fixed width text) distributed over 5 text files. I am writing a Java application that should ...
user avatar
1 vote
2 answers
319 views

How to avoid clashes in distributed computing when polling for work to do?

A big data job is split up into X partitions. The partitions are stored in a database. Status on each partition is also stored in the database and is used to ensure that each partition is only ...
user avatar
0 votes
1 answer
451 views

Application analytics - Logging with NoSQL/RDBMS, BigData [closed]

I need to architect a system for my company. My question is specific to logging. Part of the requirement is to log 2 kinds of data. A1. User events (Ex: UserA clicked on Button1, performed Action#21. ...
user avatar
3 votes
2 answers
541 views

Is manually tagging content an ideal start for eventual machine learning?

I'm putting together a mobile app that would allow users to search, rate, and upload photos of dishes at restaurants. I have no machine learning experience (just FE/BE/Mobile) so for the time being I ...
user avatar
0 votes
2 answers
1k views

The best approach for processing a large data on php

I am searching for the best approach for process a large amount data using PHP. I want to create a xml file from a database, when I execute my code the system will stop with a "PHP timeout" error, I ...
user avatar
  • 103
0 votes
1 answer
1k views

Fast storage format for huge point clouds (fast read/write)

I am working with huge point clouds (1 billion points). I need to process and display them but because of the size of the data, I can't have it all in memory at the same time. First of all, I am not ...
user avatar
  • 101
0 votes
0 answers
38 views

Can data in a rooted tree be processed in parallel?

I have data that is structured like the following: users: id | name | parent_id 1 | Bob | NULL 2 | Jan | 1 3 | Mat | 2 4 | Irene | 2 5 | Ellie | 2 6 | Laura | 5 7 | Uma | 6 ...
user avatar
0 votes
0 answers
664 views

Is MapReduce a correct framework for Extract, Transform, Load of data?

EDIT I am working on a project to update a legacy ETL infrastructure that supports a number of clients, each with a slightly different setup. Constraints that cannot be changed: Source data can ...
user avatar
2 votes
1 answer
195 views

How to make a basic database application, previously developed for 500 records, ready for 1 million records?

I have written a basic web app in PHP, using MongoDB as database engine. The app basically inserts records into database as they become available, and queries the database and displays the data as on ...
user avatar
  • 131
-1 votes
1 answer
411 views

Database design considerations for extremely large data volumes

I have an upcoming interview for a position at a company that deals with multi-petabyte scale data volumes. They're going to grill me about standard database design questions, but what are the best ...
user avatar
1 vote
1 answer
195 views

Moving from Batch to Streaming architecture using a Graph datastructure

The use case I am trying to solve is, assigning millions of users to their groups/segments. I have thousands of different criteria from which buckets of users are created. E.g. bucket criteria: All ...
user avatar
  • 111
1 vote
1 answer
883 views

Is this Big Data architecture good enough to handle many requests per second?

I want to ask for a review of my big data app plan. I haven’t much experience in that field, so every single piece of advice would be appreciated. Here is a link to a diagram of the architecture: My ...
user avatar
7 votes
4 answers
2k views

Can someone explain the technicalities of MapReduce in layman's terms?

When people talk about MapReduce you think about Google and Hadoop. But what is MapReduce itself? How does it work? I came across this blog post that tries to explain just MapReduce without Hadoop, ...
user avatar
11 votes
3 answers
15k views

How to store large amounts of _structured_ data?

The application will continuously (approximately every second) collect the location of users and store them. This data is structured. In a relational database, it would be stored as: | user | ...
user avatar
  • 1,912
3 votes
2 answers
571 views

Is Machine Learning a part of Data Science?

I am having trouble differentiating between Big Data vs Data Science. From what I've understood so far, both are buzzwords for the old statistics job. Where does machine learning fit into this?
user avatar
2 votes
1 answer
677 views

Near real-time analytics from Cassandra with frequent updates

We have an activity metrics page where users can select a date period and see other user's aggregated activity (by action) and optionally filter everything by 4 or 5 fields. Actions happen ...
user avatar
10 votes
1 answer
2k views

Partial name matching in millions of records

We have developed a web based application for name matching. It operates by breaking names into parts and the Soundex value of each part is stored in a database. The Levenshtein distance metric is ...
user avatar
  • 229
0 votes
1 answer
194 views

Best Azure Solution for Complex Search Index

I need to perform quick searches against a combination of tags while including date ranges: Example: Users who have requested notifications who did not respond to a notification sent at least 3 ...
user avatar
  • 101
3 votes
1 answer
1k views

How is one or more aggregate function implemented in most SQL engines?

In the book Database Fundamentals, Silberschatz. It is explained that aggregate functions can be calculated on the march. This make sense. What it means is that for calculating the maximun, average ...
user avatar
  • 336
1 vote
0 answers
32 views

Offsetting high usage in a multi-tenant database environment?

I'm writing a research essay over distributed cloud-based systems, and one of the topics that I'm trying to touch on is potential solutions to high database utilization in a single-database multi-...
user avatar
  • 1,317
-3 votes
1 answer
178 views

Should I use NoSQL or HDFS for storage?

I have millions of tweets currently stored in HDFS and I plan to analyze them from Spark (Data mining, text mining, Frequent Term-Based Text Clustering, Social Network Analysis) however, do not know ...
user avatar
  • 9
0 votes
1 answer
140 views

Does it make sense to store daily snapshots of data and analysis?

I just started working on some data analyzing for an e-commerce site that is moving about 250,000 packages per year. It is very seasonal and need to try to gain insights on both a macro and micro ...
user avatar
  • 111
4 votes
1 answer
1k views

Best practices for dashboard of near real-time analytics

I’m currently building a dashboard to view some analytics about the data generated by my company's product. We use MySQL as our database. The SQL queries to generate the analytics from the raw live ...
user avatar
  • 141
4 votes
3 answers
2k views

Relational vs Graph Database for (initially) moderately-sized network

We're developing an application whose data domain (or at least 90% of it) can be modeled effectively using a relational database. We've been using PostgreSQL since the beginning and have had no ...
user avatar
0 votes
2 answers
4k views

How to properly index MongoDB with ElasticSearch?

we are working on a JAVA EE project which handles huge amount of data, but has to provide full-text-search option (in hungarian language). So we started to think about what kind of architecture could ...
user avatar
  • 111
2 votes
1 answer
2k views

Feeding events to Elasticsearch, do I really need Logstash?

I'm designing an analytics system that feeds all events to Elasticsearch. The event lifecycle is as follows: Visitor does something. Custom analytics server gathers data, makes an event out of it and ...
user avatar
  • 416
2 votes
1 answer
1k views

How to architect frequent updates in No-sql db (Cassandra) - architecture

I'm storing some data in Cassandra, then after analyzing it puts in several table, I have its aggregation as daily, weekly, monthly, yearly basis. But after some time if some user reads the content I'...
user avatar
  • 121
1 vote
1 answer
144 views

BigData: time-based word count

What we are trying to do: We're trying to build a system that will count the number of unique entries for a certain timeframe. It's working ok until the data grows up or the timeframe increase, then ...
user avatar
  • 197
5 votes
1 answer
196 views

Querying large amount of data for parallel processing [closed]

I have a dataset containing list of users (around 50M). Each user has an email address, name, and some more data columns. I want to send a weekly email to those users, and the content of the email ...
user avatar
  • 159
1 vote
0 answers
94 views

Dealing with big data [closed]

I am on a project dealing with a lot of data in the form of images and videos (Data related to wind engineering). My requirement is to build a predictive algorithm based on the data I have. I have ...
user avatar
  • 216
8 votes
1 answer
1k views

quantitatively comparing AST shapes

How could one compare the shape of abstract syntax trees of similar source code programs (C, C++, Go, or anything compiled with GCC...)? I guess that plagiarism detection on source code would use ...
user avatar