Questions tagged [data]
Data are values of qualitative or quantitative variables, belonging to a set of items. Data in computing (or data processing) are represented in a structure, often tabular (represented by rows and columns), a tree (a set of nodes with parent-children relationship) or a graph structure (a set of interconnected nodes).
381
questions
1
vote
1answer
22 views
Storage and partitioning in Cosmos DB
I'm working on designing a system for reporting/storing information on hardware devices (IoT-like), and trying to figure out how that would best be structured in Azure's Cosmos DB.
Every device will ...
2
votes
2answers
54 views
Entity-Relationship language, not diagram
I have to design a database, but I don't yet know which underlying technology it will use (not just my decision). Could be SQL, could be NoSQL, could be something else.
I do have quite a few ...
1
vote
2answers
235 views
Common interface to handle different data type
I have a design question about data. I have a class "T" that has a single interface handling two data types as described below. The class T is constructed to handle one of the two data types. Users of ...
2
votes
1answer
46 views
From Oracle to Apache Parquet : how to handle eventual consistency?
I have an existing production Oracle Database. However, there are performance issues for certain kind of operations, because of the volume of the data, or the complexity of queries.
That's why I ...
3
votes
3answers
92 views
Read large JSON file and return a subset for map
To some extent, this is a wide question, because I do not know in which direction I should move.
I am using Polymaps to show markers on a map. The markers are static but the visualization depends on ...
2
votes
1answer
66 views
Techniques for handling large reporting periods in MSSQL
We have a fairly large database with some fairly large tables (100s of millions of rows). Some of those tables are reported on.
We have indexes in place to make this as fast as possible, but still ...
-3
votes
1answer
67 views
How to exchange 1T data between multiple processes?
How to exchange 1T data between multiple processes?
Requirements:
Data < 1TB. Memory > 1T.
Multiple processes (different languages) on the same machine or different machines over network
Data ...
-1
votes
1answer
75 views
How can I securely store and process user data?
I am creating a Discord Bot in Discord.Net. So first, of course this bot is not a professional project, but I still want it to look as professional as possible. And a huge part is the storage and ...
-3
votes
1answer
87 views
How to do data design without database? [closed]
I already to do research on internet about data design, they all explain how to do data design with database. But how to do data design without database? example i want to do data design for embedded ...
1
vote
1answer
71 views
C# class design for clear API wrapper
We are creating an office application in C#.
Originally, the persistent data like customer details, orders, agenda, statistics, etc. were stored in a local SqlServer database on a server. The ...
-2
votes
1answer
90 views
How to represent in a flowchart data fetched from a database
I have an existing piece of code that uses a database. In order to understand its algorithm, I try to represent it with a flow chart. In this algorithm I need to fetch some data from the database in ...
1
vote
0answers
23 views
Preventing setting regression with configuration tied to a build
I have inherited a distributed application with a lot of legacy configuration management.
Currently the application does not get it's settings updated with an application update. The flow looks ...
1
vote
0answers
35 views
Log data anonymization with ELK
I'm considering using the ELK stack for logging within our company. To comply with our data retention policy, we would like to anonymize data after one year so it cannot be traced back to an ...
1
vote
0answers
25 views
How would i design processing twitter data
I am working on a project where a user can input different criteria which will be used to fetch tweets, lets call this action as TweetAnalysis. These tweets will then be sent to another internal ...
0
votes
1answer
74 views
How do I store a trie in memory in a Spring Boot Application?
I have a spring boot application, that simply does, takes a set of characters and lists out possible english words.
Now, as everyone knows , its fairly easy, build a trie data-structure to load up a ...
5
votes
3answers
2k views
Is it anti-pattern to have inheritence in a dto?
Are data transfer objects or POJOs meant to be final or can they be extended and create hierarchies for them?
It is not clear to me if such a value class is properly designed only as a final class and ...
0
votes
3answers
296 views
Do higher order functions violate the separation of data and code principle?
I am asking here something that is really haunting for years (for real !).
I guess in a "perfect world", we would have programs made of data models and functions, and those functions manipulate data ...
-3
votes
1answer
56 views
is using redis correct to hold a dataset?
[Have restructured the question]
We have a lot of DB readers and few writers where writers may perform longer operations. This is a real-time system so the read has to be performed as efficiently as ...
-2
votes
1answer
64 views
Why does putty store sessions in the registry? [closed]
I recently wanted to copy my putty config from one laptop to another.
I figured there would be a directory somewhere which stored the IP addresses etc as some form of text file. I was surprised to ...
0
votes
1answer
95 views
Recommended Long Term Data Storage Format
I have an application (written in Go) with a lot of GPS location information. Each record consists of a latitude and longitude plus somewhere around 50 fields of various pieces of meta data, some of ...
-4
votes
1answer
52 views
How to detect that internal hard disk is removed and which files were copied? [closed]
Its easy to steal internal hdd, which do not generate sound signals to aware the owner while removing and allows Thief to copy the data.
Is there any way by which these thieves acts can be traced.
0
votes
2answers
89 views
Correct place to store semi dynamic data
What I mean by semi dynamic data are data that are expected to be changed only with product versions.
Now imagine this scenario, In version 1 of the product, I have item status : SUCCESS , FAIL
In ...
1
vote
2answers
48 views
Architecture of “tags” for contacts in a marketing system and hard-coded IDs so functions can refer to them
I have these tables: contacts, tags, contact_tags.
(contact_tags has columns contact_id and tag_id.)
Administrators can manually create new tags via an internal website.
Administrators can also ...
2
votes
1answer
45 views
Data modeling user-object permissions, between claims and metadata
I've been thinking about how to map user claims to permissions in a way that scales from a read and write perspective, probably using a stand-alone service. What we have today doesn't scale for ...
0
votes
3answers
126 views
Techniques for Cleaning/Expiring Old Data
I have a database with data that "expires" in a way -- after 10 days, the status changes from "Completed" to "Archived" and after 30 days rows still marked as "Assigned" get unassigned, for various ...
0
votes
2answers
92 views
Is it possible to transfer data with a really unique seed of a psudo random number generator
I have thinking about this idea for over 5 years and i don't have the complete technical knowledge to fully grasp the idea I'm having.
The premise of the idea is to have an extremely high base number ...
3
votes
3answers
365 views
What is the most effective algorithm to evenly distribute n points on a 2 dimensional array so they are as separated as possible?
I am trying to code a game which currently uses a matrix ( that I am calling a map). On this map are islands which can be generated n times. The purpose of the algorithm is to distribute the islands ...
2
votes
1answer
37 views
Setup pipeline to analyze data stored in web app DB
Background:
So there is a (Ruby) web app with a production Postgres DB (hosted in the cloud)
I would like to run some machine learning algorithms in a Python setting on the production data and (...
-3
votes
3answers
316 views
Is it an ok practice to keep pointers dereferenced?
For me, I prefer having all my variables, not as pointers. Is it ok to keep all heap-allocated variables dereferenced?
For example:
#include "Foo.hpp"
#include "Bar.hpp"
class Foo {
private:
Bar* ...
0
votes
1answer
70 views
Reliable data synchronisation between several systems
We have several systems that hold the same data in different databases. The requirement is to keep the data synced in as much reliable way as possible. Performance isn't a concern because we don't ...
2
votes
1answer
44 views
At what size array in Swift would you consider switching to CoreData?
I've developed a reading tracking app in Swift and for simplicity's sake I used arrays for my model. It stores some data about the books a user reads and a small image of the cover. I save them to ...
0
votes
1answer
105 views
Encoding of timestamps over cosmological time (64-bit linear vs 32-bit logarithmic)
I understand this is the place to ask questions pertaining to software design. If my question is considered to opinion-based, is there a forum or something better suited for the discussion?
...
3
votes
1answer
128 views
Arranging blocks of data for efficient memory access
suppose I have an algorthm that has N blocks of data, and an operation needs to be performed on all blocks such that requires 2 blocks to be in memory at any given time to be done for all possible ...
2
votes
1answer
106 views
Is it possible to create once loadable files?
I know very little about encryption, so I figured I'd ask the internet if it's possible to do the following (and if so, what kind of algorithm should I be looking at):
Write some code that saves data ...
2
votes
1answer
113 views
Generic way to describe data synchronization between systems?
Is there a generic way (useful formalisms) to think about data synchronization between two systems, especially in the case of slightly asymmetric roles (eg, one system contain master data, another one ...
0
votes
3answers
192 views
Getting random yet consistent data for testing
I have a big codebase yet to be covered with tests and rather limited access to the real data.
The code that needs to be covered is very data dependent. And the data is anything but shallow and ...
4
votes
1answer
89 views
Maintaining parallel libraries: Binary files access + Metadata Database ORM design
I'm dealing with a higher-level data abstraction that I would appreciate some input on.
I'm working on an application that uses a large data lake. The data lake is consisted of thousands of large ...
-1
votes
1answer
41 views
How should I update back-end data (e.g. follows, likes, etc.) which changes at an inconsistent rate?
(This is a conceptual question but as reference, I'm using Android Studio (Java) and Firebase Firestore...)
My app currently has a structure where the user can follow authors and favorite their works....
-1
votes
1answer
65 views
Formula Parsing
I have created an application to import CSV files, each CSV file that the application imports can contain different data, with new formats of the CSV files being added in the future. Rather than hard ...
3
votes
2answers
154 views
Is this data as a result of business logic or is this content as a result of GUI logic?
I'm aware of the general distinction between data and content, namely context, but here I have a case that is clearly causing a lot of confusion and debate within the company I work for.
There is a ...
1
vote
1answer
68 views
Data Matching In VBA - Best way to deal with dynamic data and user entry?
Background
I am currently building this project with VBA, just to keep in the back of your mind when thinking about my question.
Imagine 2 adjacent blocks, in Excel. The first block is made up of ...
3
votes
1answer
300 views
Design of file I/O -> processing -> file I/O system
I'm in the process of designing a new desktop application which is very different from other stuff I did before, and so I'll be happy if I could be pointed towards the right direction regarding the ...
2
votes
5answers
376 views
Which layer should have responsibility for rounding numbers?
I am developing a financial system and want to have a defined policy for rounding monetary values.
Given the following layers:
View
API
Entity Model
Persistence
If I am passing a monetary value ...
4
votes
2answers
296 views
Game Playing AI - Strategy to overcome the transposition table taking up too much memory?
I've made an engine that plays Connect Four using the standard search algorithms (minimax, alpha-beta pruning, iterative deepening, etc). I've implemented a transposition table so that the engine can ...
4
votes
1answer
181 views
Self describing data with UI
I have a use case where my team owns a web UI to aggregate data from 10+ service teams and present it to the customer in order to collect more information . This data (which is in json format) is very ...
5
votes
6answers
649 views
Is there a programming term that means property or field?
I think this question can span multiple languages, but in C# specifically there is a clear distinction between a field (ex: private int a;) and a property (ex: private int b { get; set; }). While ...
1
vote
1answer
59 views
Dictionary Coding for OLAP
Let's say we have a generic table like below:
id, name, price, quantity
20 product_x 5,00 100
20 product_y 5,00 100
20 ...
0
votes
2answers
75 views
Should / When to create javascript object to serialize data?
I'm working on a React/Redux application. I receive a data object from the api resembling this:
const dataObj = [
{
id: 1,
name: "Filo",
activity: 9,
info: {
statusCode: 4
...
0
votes
1answer
146 views
Improvements in a “data - manager - wrapper” architecture
I have programmed using the architecture below, where "client, order, piece" are just data classes, have no methods, and "managers" are the manipulators of that data, the "ClothingStore" is a wrapper ...
3
votes
1answer
117 views
Regulations about collecting data from software running at clients hardware (apps 'phoning home') [closed]
We develop an application which has now been installed on a few thousand customers sites.
Unfortunately, we have no customer data collection built into the software. This is something we want to ...
