Questions tagged [data]

Data are values of qualitative or quantitative variables, belonging to a set of items. Data in computing (or data processing) are represented in a structure, often tabular (represented by rows and columns), a tree (a set of nodes with parent-children relationship) or a graph structure (a set of interconnected nodes).

Filter by
Sorted by
Tagged with
2
votes
2answers
2k views

Data Layer vs Business Layer

I think I'm in the minority on this but would be curious to see other perspectives. A lot of times I see people talk about the Data Layer like having functions like UpdateCustomer() or Summarize(). I ...
1
vote
1answer
6k views

What is the difference between size and length? [closed]

It seems that the terms size and length are used interchangeably to describe how many bits, bytes or octets some data occupies, i.e. a length field in a data header is said to indicate the size of the ...
5
votes
1answer
190 views

Querying large amount of data for parallel processing [closed]

I have a dataset containing list of users (around 50M). Each user has an email address, name, and some more data columns. I want to send a weekly email to those users, and the content of the email ...
0
votes
1answer
484 views

Better way to transform data [closed]

Context Imagine complicated service that provides info about bank products, fees and so on. Imagine complicated UI that deals with it. UI and Service data formats are changed 1-2 times/month because ...
1
vote
0answers
90 views

Dealing with big data [closed]

I am on a project dealing with a lot of data in the form of images and videos (Data related to wind engineering). My requirement is to build a predictive algorithm based on the data I have. I have ...
4
votes
2answers
159 views

AutoIncrement Integer Column,Guid or PGuid, which to choose?

Guid has been a Global Unique Identification. Why does "PGuid" (Global Unique Identification) appear? 1) Is Guid NOT PERMENT for unique? (I sometimes wonder whether Guid is REALLY PERMENT generator ...
8
votes
3answers
3k views

How to deal with constructors in large data classes [duplicate]

Several times now I have come across the situations where you have some kind of settings class that simply contains a mass of data. Often these classes are simply not valid without at least most of ...
4
votes
2answers
930 views

Design pattern for fetching data in chunks

I am creating Qt application that uses database with huge amount of data to draw some charts. Fetching data from the database is time consuming, so is blocking the application thread or worker thread ...
0
votes
1answer
100 views

Verify That Filters on a Dataset are Exclusive

Suppose I have a data table of, say, cars with a bunch of properties for each record. Example: |Vin|Year|Make|Model|Color|Mileage| I want to allow users to define (arbitrarily complex) filters which ...
4
votes
3answers
2k views

Design pattern for large amounts of overflowing data?

Our current queues publish messages that consumed by 3rd party services with rate limits. Currently the messages are retried with exponential back-off. However there could be cases where data is ...
3
votes
3answers
112 views

Find likely candidates for visited locations along GPS track

I'm creating a visualization tool for workout data, specializing on bicycle rides. I'm looking for ways to generate more value from the data that was recorded. Since there usually is a GPS track ...
1
vote
0answers
211 views

Data Store Design for NxN Data Aggregation

I am trying to come up with a theoretical solution to an NxN problem for data aggregation and storage. As an example I have a huge amount of data that comes in via a stream. The stream sends the data ...
2
votes
3answers
260 views

Best practise for databinding when a non-data object is included with the data

I'm using .NET with Windows Forms, but I believe there may be similar concerns with other toolkits. Suppose we have a list of objects of a certain type. Using an example to make the following easier ...
3
votes
3answers
200 views

Storing great amounts of data

I was brainstorming and I found quite an interesting topic I couldn't come up with a solution for myself. How does storing excessively large amounts of data work? I mean physically, no matter how big ...
-1
votes
1answer
2k views

process large dataset c# [closed]

I have an application that process portfolios. I must aggregate 8 years of returns of aproximatelly 8K securities. (about 2016 doubles (8 years of data) for 2 dates (for comparison) 4032 for 8K ...
-2
votes
1answer
1k views

3D visualization of moment using accelerometer on Android device [closed]

Is it possible to reconstruct the movement of an Android device in 3D space using data from accelerometer? For example, I may draw a circle with my hand in the air, holding an Android device. Is ...
3
votes
2answers
295 views

How to handle multiple models that relate to a single model

First of, sorry for the vague title. In our website framework, we have modal classes to represent our database tables, that hold a rows data. I'm currently working on a stat style page, and I'm ...
1
vote
1answer
83 views

Send more data, or do client processing?

I have a conceptual problem that I want feedback on. First and foremost, the layout of my system. It contains a server, with many clients connected to it. The clients get fed data from the server at ...
9
votes
7answers
884 views

Algorithm for determining transactions among weekly data series?

I'm trying to develop a small reporting tool (with sqlite backend). I can best describe this tool as a "transaction" ledger. What I'm trying to do is keep track of "transactions" from weekly data ...
1
vote
0answers
212 views

Saving and retrieving multiple cached json/txt data files

I am working on some numerical programming and need to generate the results for a model given a variety of input parameters. Since the model takes a while to run, I was planning to save the data to ...
3
votes
2answers
727 views

How do you sync changes to the data in your database? [closed]

I am new to a team developing a very large web application. Prior to my start, the team had 0 automation in place and just did everything by hand. This was possible because they were only a team of 2. ...
1
vote
1answer
391 views

Using a distributed system as an alternative to disk storage

So, one of the differences between "main memory" and disk storage is durability -- the data on hard disks will stay accessible and unaffected even if the host machine crashes or loses power. One can ...
6
votes
1answer
863 views

What method should I use to populate a tree as I'm consuming data that is contained within the tree?

I have a tree that consists of 5 types of nodes - V, W, X, Y, and Z. V represents the root node, and a given tree has exactly one element of type V. A V contains one or more Ws, a W contains one or ...
77
votes
14answers
26k views

Are data type declarators like “int” and “char” stored in RAM when a C program executes?

When a C program is running, the data is stored on the heap or the stack. The values are stored in RAM addresses. But what about the type indicators (e.g., int or char)? Are they also stored? ...
2
votes
3answers
9k views

Is it bad to store data in JSON form or should I use mySQL? [closed]

I've been building an application and the whole thing is built around a simple data storage system. Create an array. Convert it into JSON. Add <?php die() at the beginning to prevent access. Save ...
1
vote
2answers
1k views

Avoid data manipulation by user

I have a C# program (could be any programming language) that saves data to a file on a memory (hard disk, USB drive, etc.). The program uses this data for monitoring its operation time, but it could ...
4
votes
2answers
506 views

Is this data model a list or tree?

Using python syntax with below environment diagram, teacher taught us that there are 11 trees(orange boundary) in this diagram, including leaf. It has been taught here, lists are represented as a ...
6
votes
3answers
4k views

Saving to Registry vs. CustomFile [closed]

I've been using the usual My.Settings... method when it came to saving settings for my program but I was just wondering, If I wanted to save program settings into the Registry and load those settings, ...
4
votes
4answers
250 views

Approaches for storing and analysing large amounts of time-based data

I've been asked to develop a "telemetry" application that records data generated by a hardware device, which I read every 100ms. There are approx 250 data points (32-bit values), but only a subset ...
1
vote
1answer
94 views

Benefits of this data format

I'm integrating with a bluetooth blood pressure monitor, and I've come across one of the most bizarre data formats I've ever seen. The systolic, diastolic and pulse rate (all ints, should all be ...
1
vote
2answers
1k views

One Enum vs Multiple Enums

I am developing an application where a user submits a mission and other users accept the mission. Pretty simple. I want to keep a track of the mission progress status and store it into a database. ...
0
votes
0answers
24 views

One Enum vs Multiple Enums [duplicate]

I am developing an application where a user submits a mission and other users accept the mission. Pretty simple. I want to keep a track of the mission progress status and store it into a database. ...
1
vote
1answer
539 views

File validation rules

I have an application that can accept CSV files to run some operations. The files look like: CREATE USER:username,last_name,first_name,age user1,Smith,John,23 user2,Poppins,Mary,257 There are a ...
4
votes
1answer
1k views

How to store data that is recorded with different frequency?

I'm writing a metallurgical web application, that must store data (electrical parameters) recorded at different frequencies. For example: six parameters recorded with frequency of one measure per ...
4
votes
3answers
3k views

Data format for binary data transfer

I have to make an oscilloscope. Ideally, the oscilloscope connects to a probe. This probe should be another program, and could get the data locally or from a network. This data are numbers, floats and/...
1
vote
2answers
557 views

Is there a detriment to using SQLite over a pList for persistence?

I'm making a fairly simple application and I've got a settings menu that stores times and some boolean values. I have the application storing and retrieving data already but I started to wonder if ...
0
votes
1answer
213 views

How much to encapsulate when objects being dealt with are pretty similar

As I have been answering questions related to object-oriented design and how to go about reducing the amount of classes to make sure that there is not a lot of "class clutter" and code repetition I ...
1
vote
1answer
306 views

Is a linked list an actual data structure?

I hear conflicting sides from this topic. I was under the impression that a linked list was a way to implement a data structure (stack, queue), not actually a data structure.
1
vote
0answers
84 views

How to deal with requests that don't conform to the existing data structure?

The system I'm working on has a list of assets that belong to different customers of the company. This data is saved on an SQL database. Removing irrelevant data, let's say the table is like this: +-...
1
vote
1answer
389 views

Running an algorithm against a large dataset

We have a python application which does a DFS search against data in a database. Right now, that data is small so I can hold everything in a python container but the data is going to grow and there ...
2
votes
1answer
2k views

ER Diagram Multiplicity (UML)

Schema: #=PK Employee (#NIN, fName, lName, address, DOB, sex, salary, taxCode, deptNo) Department (#deptNo, deptName, managerNIN, businessAreaNo, regionNo) Project (#projNo, projName, contractPrice, ...
4
votes
2answers
110 views

Is it a good idea to have a database row that represents an unknown value within a system [closed]

We have two systems, System A imports a list of sports events from system B, it only wants to import the events we will be providing live data for, the filter used to hide events that we are not ...
2
votes
1answer
1k views

Finding a solution for interleaving multiple data messages into one network stream (in QT framework)

I need to find a solution for high throughput network data interleaving in my QT framework based windows application. My application is based on server/client architecture. The client connects to a ...
6
votes
2answers
637 views

Bottleneck performance in ECS

I've been looking into building an entity-component-system. Basically, an entity is just an id wrapped around a struct, components are data belonging to that entity(and reference said id), and systems ...
1
vote
1answer
81 views

Does cleaning data before saving fall under the term “data sanitization”?

I understand that "data sanitization" mainly deals with cleaning user-entered data to prevent harmful effects (e.g. SQL injection), but what about other forms of cleaning data? E.g., trimming ...
1
vote
0answers
939 views

What is the difference between a single version of the truth and a single source of the truth when dealing with data?

According to Wikipedia a single version of the truth is described as "is a technical concept describing the data warehousing ideal of having either a single centralised database, or at least a ...
2
votes
1answer
1k views

Matching algorithms in context of data quality problems

I work for an organization that has lots of databases containing person information. The data quality is poor. One case was a surname I found like this (this is the worst-case scenario): Mark "...
10
votes
3answers
5k views

Where should I store test data?

I have smaller unit tests that use small snippets from real data sets. I would also like to test my program against full data sets for a multitude of reasons. The only problem is that a single real ...
0
votes
0answers
50 views

Manual repetitive conversion between file types

Let's say i have an interest in file conversions, but everything should be made by hand and i have multiple output formats (say: csv and excel). Once i get contacted by a client, i have to link ...
11
votes
2answers
1k views

Synchronization between two systems using MongoDB as changelog

We are developing two related systems. One of them (A) will be installed on our customers' machines. The remaining (B) will be used by my organization. Each system has its own database (relational) ...

1
4
5
6 7 8