Questions tagged [data]
Data are values of qualitative or quantitative variables, belonging to a set of items. Data in computing (or data processing) are represented in a structure, often tabular (represented by rows and columns), a tree (a set of nodes with parent-children relationship) or a graph structure (a set of interconnected nodes).
409
questions
1
vote
0answers
31 views
Python: Function pipeline with multiple return/input values, or use OOP? Best Practices?
I have a 'processing' function and a 'serializing' function. Currently the processor returns 4 different types of data structures to be serialized in different ways.
Looking for the best practise on ...
-1
votes
0answers
56 views
Efficiently storing a chunk of realtime data?
Let's say I'm programming a realtime stock chart and I want my program to run indefinitely without running out of memory. What might be a good way to keep the current data in memory up to a certain ...
1
vote
0answers
44 views
How to (simply) architecture a way to ingest multiple types of large files, process them, and send data in chunks to web services?
Note: All of this would be in AWS
Hi everyone,
What would you guys suggest for building something that:
Takes in several different input file types (ex: csv, json, jsonl,
xml, .gz, ...)
That can be ...
4
votes
4answers
128 views
Static data - database or code
I work on small to medium, database driven, line of business applications. What I usually find, especially in older systems, is most of the data lives in the database. By data I mean stuff like a ...
0
votes
2answers
80 views
Appropriate storage to store all keyboard and mouse events on my computer?
I'd like to build detail analytics of my computer usage to try and detect patterns and improvement opportunities. So I'm building a software to record every mouse click, keyboard event and window ...
-3
votes
1answer
45 views
Difference between read and write access (data, IoT)
i am a lawyer and currently writing an essay about data access in ioT environments. I often stumble over the distinction between read access and write access. Could someone provide me with an example, ...
-3
votes
2answers
97 views
Sending JSON in body of an email
Basically, in our scenario, data has to be sent from a WordPress form to Dynamics 365. My current approach is putting it into an email body in JSON format, that is then extracted and further processed ...
0
votes
2answers
105 views
How can I associate algorithm-specific data to the objects it works with, efficiently yet cleanly?
I'm writing in C++, but this problem applies to most non-high-level languages, and possibly some high-level ones as well.
I have a graph of heterogeneous nodes. The graph can be instantiated by ...
3
votes
12answers
653 views
Why do or should programmers save data in text based formats like JSON or XML instead of binary?
I see a lot of reasons to use binary over text-based formats.
With binary, I find it a lot easier. I can use fread(data, sizeof(struct DataStruct), 1, fileptr) to read, or fwrite to write data. I ...
-2
votes
2answers
239 views
Where to put data generating functions for unit tests
I am writing a unit-test framework (for Matlab/Octave, but that does not matter) which also supports property based testing. Thus, I need some functions which generate random data which are tailored ...
-1
votes
1answer
53 views
Best practices for data manipulation to use in admin dashboard application?
I am trying to create admin dashboard to track sales and inventory. Using React front end and Django rest backend API.
Right now I am pulling sales and would like to make some charts.
What's the ...
-7
votes
1answer
64 views
How would one 'falsify' data collected of you by companies using software? [closed]
Since we live in a world where basically all your web activity is being tracked by some company or your ISP. I was wondering how to 'falsify' the data they collect.
For instance by having software go ...
0
votes
1answer
47 views
IOT data interpretation
This is a more architectural question and I am hoping to get more ideas out of this thread. I have a computer vision model detecting daily life activities and sending data to the server every second. ...
0
votes
1answer
75 views
how to store the data for function f(time,length)
I have a function which has discrete time step ,discrete length from origin . These two give height of the function. For example at time 1.2sec , and length 5.5cm from origin height is 10cm. The step ...
0
votes
4answers
169 views
System Design: Efficient way to import hundreds of different CSV formats
OVERVIEW
We have multiple data providers who provide us information from a large set of data sources. These end data sources provide data in CSV format.
Since, the CSV formats are not consistent ...
1
vote
1answer
99 views
An effective way of finding regressions
There is a rather large data flow engine - more than 2000 different flow definitions of "what to do with inbound data". The engine deals with various data formats (flat-file, CSV, JSON, XML, ...
3
votes
2answers
176 views
How to ensure data consistency between 2 microservices while both having write permissions
I'm currently building my first ever microservice using NodeJS, but I stumbled upon a problem.
TL;DR
~ How to ensure data consistency between 2 microservices while both having write permissions
Setup
...
0
votes
2answers
335 views
Terminology: Classes that contain only methods [closed]
I am studying the programming language Kotlin, and I just came across the idea of a Data Class. I have a background in Java programming where classes can have fields and methods. I have heard ...
1
vote
1answer
97 views
When modules written in different languages interact, is exchanged data always serialized?
I have module written in Python and I would like to write function in e.g. Go but problem is that data needed for computations is often more than 100MB. My question is: is data always serialized and ...
-5
votes
1answer
38 views
How is data stored prior to downloading?
I go to several sites that offer their data in XML, JSON, CSV formats. As shown here,
https://www.space.org/basicspacedata/query/class/boxscore/format/csv, I choose the format. Question is how is ...
0
votes
2answers
132 views
How to minimize size of data given the need for a null pointer?
So I don't know how to formulate this clearly, as it's quite confusing. Hence the reason for asking this question, to overcome the confusion.
I will do my best to make it clear, but part of the ...
-4
votes
1answer
40 views
What to call a system that gives users customized access to data?
What would be the proper way to describe a system whose sole purpose it is to give users access to large amounts of data in a structured way according to their own criteria?
Like say for example ...
1
vote
0answers
26 views
Rolling Analytics - Dates and Dropoffs
I would like to generate analytics on a per-user basis on how many times they've viewed a particular page for 7 days, and how many times they've viewed a page during their lifetime as a user.
This ...
1
vote
2answers
106 views
How important is the data to assess code coverage?
per the title how important is the data in assessing code coverage?
As background, let's say that you're given 20% of the entire dataset to help trace the different pathways each row of data goes ...
2
votes
2answers
113 views
How to represent a data model?
Often enough, the terms data model and data format are used interchangeably, but here I disagree. Let's start with the simpler one, the data format. I don't know exact definitions, but the data format ...
2
votes
2answers
974 views
Storing unchangable data in database vs in code
Hi I am trying to create a role based dynamic route authorization system on my NodeJS ExpressJS powered API server.
Scenario:
There will be some roles like Admin, StandardUser etc. and they can't be ...
-3
votes
1answer
70 views
Can TLV theoretically be used for browsers?
Compared with HTML (Hypertext Markup Language), TLV (Tag-Length-Value) also can be used for a data representation. If TLV is designed smart enough, it can represent hierarchies and trees with no ...
3
votes
2answers
78 views
How to synchronize changes between Data Engineers and Designers?
I'm processing data and writing in a database, that is used by my colleague, a designer, working on data viz.
How can I efficiently, say, add a new column for my colleague to use in their ...
0
votes
1answer
97 views
What are differences in application architecture in frontend and backend applications?
I'm pretty experienced with developing both frontend as well as backend applications, with a variety of programming languages and frameworks. I know the problem space and concepts involved in both ...
1
vote
5answers
350 views
How do B-Trees used in databases sort data based on different fields?
I know majority of databases uses B-Trees, and I can see how using a balanced binary tree will give fast sort times, for ordering by ID or whatever else the primary key is; but how are databases able ...
1
vote
1answer
32 views
Storage and partitioning in Cosmos DB
I'm working on designing a system for reporting/storing information on hardware devices (IoT-like), and trying to figure out how that would best be structured in Azure's Cosmos DB.
Every device will ...
2
votes
3answers
114 views
Entity-Relationship language, not diagram
I have to design a database, but I don't yet know which underlying technology it will use (not just my decision). Could be SQL, could be NoSQL, could be something else.
I do have quite a few ...
1
vote
2answers
301 views
Common interface to handle different data type
I have a design question about data. I have a class "T" that has a single interface handling two data types as described below. The class T is constructed to handle one of the two data types. Users of ...
2
votes
1answer
613 views
From Oracle to Apache Parquet : how to handle eventual consistency?
I have an existing production Oracle Database. However, there are performance issues for certain kind of operations, because of the volume of the data, or the complexity of queries.
That's why I ...
3
votes
3answers
220 views
Read large JSON file and return a subset for map
To some extent, this is a wide question, because I do not know in which direction I should move.
I am using Polymaps to show markers on a map. The markers are static but the visualization depends on ...
2
votes
1answer
73 views
Techniques for handling large reporting periods in MSSQL
We have a fairly large database with some fairly large tables (100s of millions of rows). Some of those tables are reported on.
We have indexes in place to make this as fast as possible, but still ...
-3
votes
1answer
70 views
How to exchange 1T data between multiple processes?
How to exchange 1T data between multiple processes?
Requirements:
Data < 1TB. Memory > 1T.
Multiple processes (different languages) on the same machine or different machines over network
Data ...
-1
votes
1answer
84 views
How can I securely store and process user data?
I am creating a Discord Bot in Discord.Net. So first, of course this bot is not a professional project, but I still want it to look as professional as possible. And a huge part is the storage and ...
-3
votes
1answer
145 views
How to do data design without database? [closed]
I already to do research on internet about data design, they all explain how to do data design with database. But how to do data design without database? example i want to do data design for embedded ...
1
vote
1answer
89 views
C# class design for clear API wrapper
We are creating an office application in C#.
Originally, the persistent data like customer details, orders, agenda, statistics, etc. were stored in a local SqlServer database on a server. The ...
-2
votes
1answer
3k views
How to represent in a flowchart data fetched from a database
I have an existing piece of code that uses a database. In order to understand its algorithm, I try to represent it with a flow chart. In this algorithm I need to fetch some data from the database in ...
1
vote
0answers
23 views
Preventing setting regression with configuration tied to a build
I have inherited a distributed application with a lot of legacy configuration management.
Currently the application does not get it's settings updated with an application update. The flow looks ...
1
vote
0answers
69 views
Log data anonymization with ELK
I'm considering using the ELK stack for logging within our company. To comply with our data retention policy, we would like to anonymize data after one year so it cannot be traced back to an ...
1
vote
0answers
25 views
How would i design processing twitter data
I am working on a project where a user can input different criteria which will be used to fetch tweets, lets call this action as TweetAnalysis. These tweets will then be sent to another internal ...
1
vote
1answer
628 views
How do I store a trie in memory in a Spring Boot Application?
I have a spring boot application, that simply does, takes a set of characters and lists out possible english words.
Now, as everyone knows , its fairly easy, build a trie data-structure to load up a ...
8
votes
3answers
4k views
Is it anti-pattern to have inheritence in a dto?
Are data transfer objects or POJOs meant to be final or can they be extended and create hierarchies for them?
It is not clear to me if such a value class is properly designed only as a final class and ...
0
votes
3answers
323 views
Do higher order functions violate the separation of data and code principle?
I am asking here something that is really haunting for years (for real !).
I guess in a "perfect world", we would have programs made of data models and functions, and those functions manipulate data ...
-3
votes
1answer
61 views
is using redis correct to hold a dataset?
[Have restructured the question]
We have a lot of DB readers and few writers where writers may perform longer operations. This is a real-time system so the read has to be performed as efficiently as ...
-2
votes
1answer
76 views
Why does putty store sessions in the registry? [closed]
I recently wanted to copy my putty config from one laptop to another.
I figured there would be a directory somewhere which stored the IP addresses etc as some form of text file. I was surprised to ...
0
votes
1answer
238 views
Recommended Long Term Data Storage Format
I have an application (written in Go) with a lot of GPS location information. Each record consists of a latitude and longitude plus somewhere around 50 fields of various pieces of meta data, some of ...
