Questions tagged [data]

Data are values of qualitative or quantitative variables, belonging to a set of items. Data in computing (or data processing) are represented in a structure, often tabular (represented by rows and columns), a tree (a set of nodes with parent-children relationship) or a graph structure (a set of interconnected nodes).

Filter by
Sorted by
Tagged with
1 vote
3 answers
1k views

Why do CSV file formats normally use quoting instead of escaping?

There are a lot of variants of the CSV "standard" (or lack thereof). I've never personally see any that use an escape character (like \) instead of surrounding each field with double quotes. Instead ...
1 vote
3 answers
806 views

Repository w/ MVVM Pattern: Data Format Check, Where/When, Best Practice?

TL; DR Where/When to check data format in MVVM + Repository Pattern is the best practice? Any suggestion or theory? The Scene Our team is working on an Android Project. We have a MVVM+repo structure: ...
17 votes
4 answers
4k views

How should I store readonly data to deploy with my application?

I'm developing a desktop application, and this application requires some information to run, but it doesn't change any of this information (the data must be loaded on every execution of the app, but ...
0 votes
1 answer
205 views

Data Matching In VBA - Best way to deal with dynamic data and user entry?

Background I am currently building this project with VBA, just to keep in the back of your mind when thinking about my question. Imagine 2 adjacent blocks, in Excel. The first block is made up of ...
1 vote
1 answer
592 views

Transform scanned PDF documents into Excel using OCR

I have a task that is to convert cable schedules into an Excel spreadsheet. I have tried a few different OCR (Optical Character Recognition) things such as websites, code in R (tesseract), javascript, ...
1 vote
1 answer
90 views

Persisting data in a mobile app backed by a rest API back-end, advices [closed]

So I am working on this mobile application using MAUI, which is backed by a Rest API in ASP.NET Core. I want to be able to use the app offline, and the user can create/edit some data. This data is ...
2 votes
1 answer
241 views

Data modeling user-object permissions, between claims and metadata

I've been thinking about how to map user claims to permissions in a way that scales from a read and write perspective, probably using a stand-alone service. What we have today doesn't scale for ...
7 votes
3 answers
1k views

Prevent analytics from crashing the production database

At the core of my business (say, an online store) is a Postgres database that stores products and transactions. During day-to-day operations, the load on the database is not heavy. However, there is ...
1 vote
1 answer
102 views

How to approach converting to and from kafka messages?

I would like to have a series of small stand-alone services that would either consume a Kafka topic and output the data into a different system or the reverse: receive data from a system and produce ...
0 votes
1 answer
103 views

Storing Data For Consumption in Python

I have a program which needs some constant data, in JSON-like format. However, this data only needs to be consumed by my Python program, and by making it Python code, I can include types like datetime....
10 votes
1 answer
4k views

"Reactive programming is programming with asynchronous data streams." is this true?

I was reading about Reactive programming, and came across this article The introduction to Reactive Programming you've been missing with the quote I put in the question title: Reactive programming ...
1 vote
3 answers
325 views

How do you test whether schema changes will break services?

I maintain a relational database of sorts, and occasionally a feature request requires a schema change (eg. add a new column, etc...). However, a number of "services" consume this data. It's ...
0 votes
0 answers
153 views

Transactionally update a set of files in s3

I have a bucket with several csv files at differing paths. The problem is that I want to update these csv files transactionally. The s3 bucket should represent the "latest" consistent set of ...
0 votes
2 answers
137 views

Does Automated Pipeline Mean CI/CD Pipeline?

I am not sure if I have been using the term wrongly (and including it in my CV), so some inputs from the community will be appreciated. I am not a DevOps, but a noob machine learning engineer. So I ...
0 votes
4 answers
450 views

How to safely store application data on shutdown?

I am writing a Windows Service using the Windows Background Worker Service template. Some data that I track needs to persist between application instances (i.e. after shutdown and restart). Initially, ...
0 votes
3 answers
399 views

Perfomance of 8 bit operations on 64 bit architechture

I am usually a Python / Database programmer, and I am considering using C for a problem. I have a set of sequences, 8 characters long with 4 possible characters. My problem involves combining sets of ...
2 votes
2 answers
81 views

At what size array in Swift would you consider switching to CoreData?

I've developed a reading tracking app in Swift and for simplicity's sake I used arrays for my model. It stores some data about the books a user reads and a small image of the cover. I save them to ...
0 votes
2 answers
81 views

How do I control many datasets that are related and can change independently?

Background: I'm a scientist working with a small team on data analytics. No one on our team has any experience with software engineering, though I'm trying to change that. Suppose I have an array A ...
1 vote
0 answers
102 views

Design patterns for versioning steps across data/workflow pipelines?

I'm sure this has been touched upon by a number of questions, but I'm struggling on drawing the boundaries between code, data and configuration versions when working with a large DAG (think airflow or ...
-3 votes
2 answers
83 views

Optimal variable-time logging of a real-time data stream

Cross-posted from stats stackexchange Say I have a logging utility in my application that I use for recording timestamped diagnostic log messages. I want to add tracking of some performance metrics to ...
5 votes
7 answers
1k views

Is there a programming term that means property or field?

I think this question can span multiple languages, but in C# specifically there is a clear distinction between a field (ex: private int a;) and a property (ex: private int b { get; set; }). While ...
0 votes
1 answer
81 views

load testing - spike vs stress to determine performance

During the course of my load testing journey, I'm doing mostly spike test on the SUT(System Under Test). We gradually dial up the number of users hitting the server, from 10, 100, 1000. And we log the ...
1 vote
1 answer
118 views

What is a common practice to refer to other datas within a data oriented design?

This question has been in my mind for a while now, especially in the context of high performance, interactive 3d applications. Just want to find out what is the general practices in DoD for ...
3 votes
2 answers
197 views

How to propagate seldomly changing data in a distributed system

We store data – some kind of metadata like car manufacturers or types of computer parts – in one of our applications. This data changes rarely. Let's say it changes once every two weeks. This data is ...
-1 votes
1 answer
41 views

Scalable Web Data Service Approach

I'm looking to build a service that interacts with an existing web application's database. The service would retrieve data from an api for which the web application users own an account and warehouse/...
0 votes
1 answer
111 views

How is model reconstructed with event sourcing in CQRS principle?

I'm trying to understand the CQRS design. As far as I know now, every time a change to a model is saved, the (state) change is stored as a separate record. When reading a model, the model is ...
0 votes
1 answer
116 views

UML design for data analytics of aggregated Survey Data

First and foremost, let me say that I do not have a Software Engineering background . I need help from the community as I have been assigned to create a UML (Unified Modeling Language) design from ...
1 vote
2 answers
417 views

Whose job actually is to process data in the database to become something like a chart in frontend?

I'm new here, so please be gentle. I've been having issues at my workplace about the front-end team demanding me as a backend to provide every piece of data that has to be displayed in the front end. ...
0 votes
2 answers
118 views

Data sharing across processes [closed]

I have a C# .NET app that downloads a users Tweets using LinqToTwitter. This is for TV broadcast - the client wants to show Tweets on air. We have two other apps that need to access these tweets. One ...
1 vote
2 answers
74 views

If I want to add 'period' in the database (one month), and would like to make calculations with the first day of the period. What would I have to do? [closed]

I have a question. I want to add 'period' in the database. A period would always be 1 month. How Would I be able to make calculations with the period? For example I want to calculate the following: A =...
5 votes
2 answers
341 views

How did the IEEE 754 standard choose how many bits to allocate to each field (i.e. sign, exponent, mantissa)?

I've been reading about the IEEE 754 standard, but can't understand how or why they chose those exact number of bits for the exponent and mantissa. For example, why 8 bits for the exponent and not 7 ...
0 votes
3 answers
289 views

What's stopping an app from using another platform's storage, for free?

Let me clarify on a bit of an abstract question. I'm about to develop an app, through which users can capture and store images. Commercially, the main cost here that springs to mind is storage cost. ...
2 votes
3 answers
201 views

Eliminating combinations based on user input

I am stuck with this particular problem. To give context to the problem, I am developing a mobile app which helps with loading dangerous goods onto a truck. Ignore the size and weight of the dangerous ...
3 votes
1 answer
790 views

Should reflection be part of design?

I'm currently designing a somewhat large program that will involve the simulation of math/physics models and collection of data (I have not implemented any code yet). One of the main problems I'm ...
2 votes
1 answer
239 views

User (Actor) data shared across Domains / Microservices

I have read through a number of posts on Software Engineering but am unable to find a suitable solution for the problem my team and I are trying to solve. We are building a system where we have ...
10 votes
1 answer
766 views

Does current evidence support the adoption of Contextual over Canonical Data Models?

The "canonical" idea is pervasive in software; patterns like Canonical Model, Canonical Schema, Canonical Data Model and so on, seem to come up again and again in development. Like many developers, I'...
1 vote
0 answers
812 views

Best way to store key-value pairs of different types in c++?

I'm reading values from a (PE) binary file. The values have a known length and position in the file and are stored with no padding (right next to each other). They store various types (int, long, ...
0 votes
3 answers
171 views

Can you suggest ways to assure client data confidentiality in a SaaS product managed by yourself?

I am designing a multitenant SaaS platform where the database will hold information for different clients. The application consists of an Angular frontend and a REST API Backend. Some of the clients ...
-3 votes
1 answer
212 views

Adding rows and columns simultaneously to pandas dataframe

I am new to Python (~ 1 month) and am having a hard time figuring out a clean, fast way to simultaneously append rows and columns to a dataframe automatically. As a representative example, I will ...
0 votes
1 answer
2k views

Decoupled architecture between business and data layers in Spring JPA / Hibernate

I'm using Spring Boot with JPA / Hibernate and I'm trying to decouple the business layer from the data layer. I would like to be able to (relatively easily) switch from a relational database to a rdf ...
2 votes
2 answers
1k views

Storing unchangeable data in database vs in code

Hi I am trying to create a role based dynamic route authorization system on my NodeJS ExpressJS powered API server. Scenario: There will be some roles like Admin, StandardUser etc. and they can't be ...
2 votes
2 answers
3k views

From Oracle to Apache Parquet : how to handle eventual consistency?

I have an existing production Oracle Database. However, there are performance issues for certain kind of operations, because of the volume of the data, or the complexity of queries. That's why I ...
0 votes
2 answers
421 views

Share data between users without a database, php [closed]

How would I create for example a live chat, where you "post" a piece of text and it is displayed to other users in a "chat room", without the need of permanently storing it. My ...
0 votes
2 answers
2k views

Best practice for creating defined versions in git

In my company we have a 'production pipeline', which is a bunch of code which we use to produce data, with the data later shipped to customers. The code is still being developed, bugs get fixed etc. ...
2 votes
2 answers
2k views

Duplication of data vs loose coupling

Currently, I'm working with a health management application, let's call it application A. It's a partly prebuilt generic application that we are extending on. The main entities are journals, events, ...
0 votes
1 answer
89 views

Data Class: spread boilerplate for each different database over multiple class copies, or add complexity to original class?

I'm creating a prototype mobile app that will store data locally in an sqLite database and also on the cloud (CloudKit for now). The cloud storage uses a certain way of storing the data, and sqlite ...
0 votes
1 answer
122 views

how to store the data for function f(time,length)

I have a function which has discrete time step ,discrete length from origin . These two give height of the function. For example at time 1.2sec , and length 5.5cm from origin height is 10cm. The step ...
1 vote
1 answer
182 views

Integration with stripe, have a single source of truth or have info of the user status in our DB

So We're integrating our application with stripe and we have different products There are 2 options : Have stripe as a single source of truth, anytime we need to check the permissions of a user we ...
1 vote
1 answer
2k views

How do I store a trie in memory in a Spring Boot Application?

I have a spring boot application, that simply does, takes a set of characters and lists out possible english words. Now, as everyone knows , its fairly easy, build a trie data-structure to load up a ...
1 vote
1 answer
472 views

SQL Foreign Key Relationships and External Data

I work for a company that wants an external system built outside of their core system. I can't share an SQL database, and can't store data in their database. So I am forced to maintain my own SQL ...

1
2 3 4 5
10