Questions tagged [data]
Data are values of qualitative or quantitative variables, belonging to a set of items. Data in computing (or data processing) are represented in a structure, often tabular (represented by rows and columns), a tree (a set of nodes with parent-children relationship) or a graph structure (a set of interconnected nodes).
397
questions
-1
votes
0answers
8 views
how to store the data for function f(time,length)
I have a function which has discrete time step ,discrete length from origin . These two give height of the function. For example at time 1.2sec , and length 5.5cm from origin height is 10cm. The step ...
0
votes
3answers
146 views
System Design: Efficient way to import hundreds of different CSV formats
OVERVIEW
We have multiple data providers who provide us information from a large set of data sources. These end data sources provide data in CSV format.
Since, the CSV formats are not consistent ...
-2
votes
0answers
27 views
How to handle big data analytics on web?
I'm working on a project based on django (python) and mongodb as database that shows some analytics(figures produced from calculated data from database) based on some csv data. Currently my local ...
1
vote
1answer
96 views
An effective way of finding regressions
There is a rather large data flow engine - more than 2000 different flow definitions of "what to do with inbound data". The engine deals with various data formats (flat-file, CSV, JSON, XML, ...
3
votes
2answers
164 views
How to ensure data consistency between 2 microservices while both having write permissions
I'm currently building my first ever microservice using NodeJS, but I stumbled upon a problem.
TL;DR
~ How to ensure data consistency between 2 microservices while both having write permissions
Setup
...
0
votes
2answers
253 views
Terminology: Classes that contain only methods [closed]
I am studying the programming language Kotlin, and I just came across the idea of a Data Class. I have a background in Java programming where classes can have fields and methods. I have heard ...
1
vote
1answer
97 views
When modules written in different languages interact, is exchanged data always serialized?
I have module written in Python and I would like to write function in e.g. Go but problem is that data needed for computations is often more than 100MB. My question is: is data always serialized and ...
-5
votes
1answer
36 views
How is data stored prior to downloading?
I go to several sites that offer their data in XML, JSON, CSV formats. As shown here,
https://www.space.org/basicspacedata/query/class/boxscore/format/csv, I choose the format. Question is how is ...
0
votes
2answers
131 views
How to minimize size of data given the need for a null pointer?
So I don't know how to formulate this clearly, as it's quite confusing. Hence the reason for asking this question, to overcome the confusion.
I will do my best to make it clear, but part of the ...
-4
votes
1answer
39 views
What to call a system that gives users customized access to data?
What would be the proper way to describe a system whose sole purpose it is to give users access to large amounts of data in a structured way according to their own criteria?
Like say for example ...
1
vote
0answers
24 views
Rolling Analytics - Dates and Dropoffs
I would like to generate analytics on a per-user basis on how many times they've viewed a particular page for 7 days, and how many times they've viewed a page during their lifetime as a user.
This ...
1
vote
2answers
106 views
How important is the data to assess code coverage?
per the title how important is the data in assessing code coverage?
As background, let's say that you're given 20% of the entire dataset to help trace the different pathways each row of data goes ...
1
vote
2answers
98 views
How to represent a data model?
Often enough, the terms data model and data format are used interchangeably, but here I disagree. Let's start with the simpler one, the data format. I don't know exact definitions, but the data format ...
2
votes
2answers
971 views
Storing unchangable data in database vs in code
Hi I am trying to create a role based dynamic route authorization system on my NodeJS ExpressJS powered API server.
Scenario:
There will be some roles like Admin, StandardUser etc. and they can't be ...
-3
votes
1answer
69 views
Can TLV theoretically be used for browsers?
Compared with HTML (Hypertext Markup Language), TLV (Tag-Length-Value) also can be used for a data representation. If TLV is designed smart enough, it can represent hierarchies and trees with no ...
3
votes
2answers
78 views
How to synchronize changes between Data Engineers and Designers?
I'm processing data and writing in a database, that is used by my colleague, a designer, working on data viz.
How can I efficiently, say, add a new column for my colleague to use in their ...
0
votes
1answer
94 views
What are differences in application architecture in frontend and backend applications?
I'm pretty experienced with developing both frontend as well as backend applications, with a variety of programming languages and frameworks. I know the problem space and concepts involved in both ...
1
vote
5answers
251 views
How do B-Trees used in databases sort data based on different fields?
I know majority of databases uses B-Trees, and I can see how using a balanced binary tree will give fast sort times, for ordering by ID or whatever else the primary key is; but how are databases able ...
1
vote
1answer
32 views
Storage and partitioning in Cosmos DB
I'm working on designing a system for reporting/storing information on hardware devices (IoT-like), and trying to figure out how that would best be structured in Azure's Cosmos DB.
Every device will ...
2
votes
3answers
94 views
Entity-Relationship language, not diagram
I have to design a database, but I don't yet know which underlying technology it will use (not just my decision). Could be SQL, could be NoSQL, could be something else.
I do have quite a few ...
1
vote
2answers
277 views
Common interface to handle different data type
I have a design question about data. I have a class "T" that has a single interface handling two data types as described below. The class T is constructed to handle one of the two data types. Users of ...
2
votes
1answer
352 views
From Oracle to Apache Parquet : how to handle eventual consistency?
I have an existing production Oracle Database. However, there are performance issues for certain kind of operations, because of the volume of the data, or the complexity of queries.
That's why I ...
3
votes
3answers
155 views
Read large JSON file and return a subset for map
To some extent, this is a wide question, because I do not know in which direction I should move.
I am using Polymaps to show markers on a map. The markers are static but the visualization depends on ...
2
votes
1answer
72 views
Techniques for handling large reporting periods in MSSQL
We have a fairly large database with some fairly large tables (100s of millions of rows). Some of those tables are reported on.
We have indexes in place to make this as fast as possible, but still ...
-3
votes
1answer
70 views
How to exchange 1T data between multiple processes?
How to exchange 1T data between multiple processes?
Requirements:
Data < 1TB. Memory > 1T.
Multiple processes (different languages) on the same machine or different machines over network
Data ...
-1
votes
1answer
77 views
How can I securely store and process user data?
I am creating a Discord Bot in Discord.Net. So first, of course this bot is not a professional project, but I still want it to look as professional as possible. And a huge part is the storage and ...
-3
votes
1answer
123 views
How to do data design without database? [closed]
I already to do research on internet about data design, they all explain how to do data design with database. But how to do data design without database? example i want to do data design for embedded ...
1
vote
1answer
84 views
C# class design for clear API wrapper
We are creating an office application in C#.
Originally, the persistent data like customer details, orders, agenda, statistics, etc. were stored in a local SqlServer database on a server. The ...
-2
votes
1answer
2k views
How to represent in a flowchart data fetched from a database
I have an existing piece of code that uses a database. In order to understand its algorithm, I try to represent it with a flow chart. In this algorithm I need to fetch some data from the database in ...
1
vote
0answers
23 views
Preventing setting regression with configuration tied to a build
I have inherited a distributed application with a lot of legacy configuration management.
Currently the application does not get it's settings updated with an application update. The flow looks ...
1
vote
0answers
59 views
Log data anonymization with ELK
I'm considering using the ELK stack for logging within our company. To comply with our data retention policy, we would like to anonymize data after one year so it cannot be traced back to an ...
1
vote
0answers
25 views
How would i design processing twitter data
I am working on a project where a user can input different criteria which will be used to fetch tweets, lets call this action as TweetAnalysis. These tweets will then be sent to another internal ...
2
votes
1answer
379 views
How do I store a trie in memory in a Spring Boot Application?
I have a spring boot application, that simply does, takes a set of characters and lists out possible english words.
Now, as everyone knows , its fairly easy, build a trie data-structure to load up a ...
5
votes
3answers
3k views
Is it anti-pattern to have inheritence in a dto?
Are data transfer objects or POJOs meant to be final or can they be extended and create hierarchies for them?
It is not clear to me if such a value class is properly designed only as a final class and ...
0
votes
3answers
315 views
Do higher order functions violate the separation of data and code principle?
I am asking here something that is really haunting for years (for real !).
I guess in a "perfect world", we would have programs made of data models and functions, and those functions manipulate data ...
-3
votes
1answer
58 views
is using redis correct to hold a dataset?
[Have restructured the question]
We have a lot of DB readers and few writers where writers may perform longer operations. This is a real-time system so the read has to be performed as efficiently as ...
-2
votes
1answer
70 views
Why does putty store sessions in the registry? [closed]
I recently wanted to copy my putty config from one laptop to another.
I figured there would be a directory somewhere which stored the IP addresses etc as some form of text file. I was surprised to ...
0
votes
1answer
171 views
Recommended Long Term Data Storage Format
I have an application (written in Go) with a lot of GPS location information. Each record consists of a latitude and longitude plus somewhere around 50 fields of various pieces of meta data, some of ...
-4
votes
1answer
75 views
How to detect that internal hard disk is removed and which files were copied? [closed]
Its easy to steal internal hdd, which do not generate sound signals to aware the owner while removing and allows Thief to copy the data.
Is there any way by which these thieves acts can be traced.
0
votes
2answers
124 views
Correct place to store semi dynamic data
What I mean by semi dynamic data are data that are expected to be changed only with product versions.
Now imagine this scenario, In version 1 of the product, I have item status : SUCCESS , FAIL
In ...
1
vote
2answers
50 views
Architecture of “tags” for contacts in a marketing system and hard-coded IDs so functions can refer to them
I have these tables: contacts, tags, contact_tags.
(contact_tags has columns contact_id and tag_id.)
Administrators can manually create new tags via an internal website.
Administrators can also ...
1
vote
1answer
56 views
Data modeling user-object permissions, between claims and metadata
I've been thinking about how to map user claims to permissions in a way that scales from a read and write perspective, probably using a stand-alone service. What we have today doesn't scale for ...
0
votes
3answers
131 views
Techniques for Cleaning/Expiring Old Data
I have a database with data that "expires" in a way -- after 10 days, the status changes from "Completed" to "Archived" and after 30 days rows still marked as "Assigned" get unassigned, for various ...
0
votes
2answers
94 views
Is it possible to transfer data with a really unique seed of a psudo random number generator
I have thinking about this idea for over 5 years and i don't have the complete technical knowledge to fully grasp the idea I'm having.
The premise of the idea is to have an extremely high base number ...
3
votes
3answers
772 views
What is the most effective algorithm to evenly distribute n points on a 2 dimensional array so they are as separated as possible?
I am trying to code a game which currently uses a matrix ( that I am calling a map). On this map are islands which can be generated n times. The purpose of the algorithm is to distribute the islands ...
2
votes
1answer
37 views
Setup pipeline to analyze data stored in web app DB
Background:
So there is a (Ruby) web app with a production Postgres DB (hosted in the cloud)
I would like to run some machine learning algorithms in a Python setting on the production data and (...
-3
votes
3answers
565 views
Is it an ok practice to keep pointers dereferenced?
For me, I prefer having all my variables, not as pointers. Is it ok to keep all heap-allocated variables dereferenced?
For example:
#include "Foo.hpp"
#include "Bar.hpp"
class Foo {
private:
Bar* ...
0
votes
1answer
74 views
Reliable data synchronisation between several systems
We have several systems that hold the same data in different databases. The requirement is to keep the data synced in as much reliable way as possible. Performance isn't a concern because we don't ...
2
votes
1answer
51 views
At what size array in Swift would you consider switching to CoreData?
I've developed a reading tracking app in Swift and for simplicity's sake I used arrays for my model. It stores some data about the books a user reads and a small image of the cover. I save them to ...
0
votes
1answer
212 views
Encoding of timestamps over cosmological time (64-bit linear vs 32-bit logarithmic)
I understand this is the place to ask questions pertaining to software design. If my question is considered to opinion-based, is there a forum or something better suited for the discussion?
...
