Questions tagged [data]

Data are values of qualitative or quantitative variables, belonging to a set of items. Data in computing (or data processing) are represented in a structure, often tabular (represented by rows and columns), a tree (a set of nodes with parent-children relationship) or a graph structure (a set of interconnected nodes).

Filter by
Sorted by
Tagged with
7 votes
3 answers
1k views

Prevent analytics from crashing the production database

At the core of my business (say, an online store) is a Postgres database that stores products and transactions. During day-to-day operations, the load on the database is not heavy. However, there is ...
firstname gklsodascb's user avatar
1 vote
1 answer
67 views

How to approach converting to and from kafka messages?

I would like to have a series of small stand-alone services that would either consume a Kafka topic and output the data into a different system or the reverse: receive data from a system and produce ...
mparada's user avatar
  • 11
0 votes
1 answer
93 views

Storing Data For Consumption in Python

I have a program which needs some constant data, in JSON-like format. However, this data only needs to be consumed by my Python program, and by making it Python code, I can include types like datetime....
pigrammer's user avatar
  • 111
1 vote
3 answers
188 views

How do you test whether schema changes will break services?

I maintain a relational database of sorts, and occasionally a feature request requires a schema change (eg. add a new column, etc...). However, a number of "services" consume this data. It's ...
MYK's user avatar
  • 343
0 votes
0 answers
78 views

Transactionally update a set of files in s3

I have a bucket with several csv files at differing paths. The problem is that I want to update these csv files transactionally. The s3 bucket should represent the "latest" consistent set of ...
Jeremy Fisher's user avatar
0 votes
2 answers
117 views

Does Automated Pipeline Mean CI/CD Pipeline?

I am not sure if I have been using the term wrongly (and including it in my CV), so some inputs from the community will be appreciated. I am not a DevOps, but a noob machine learning engineer. So I ...
Della's user avatar
  • 121
0 votes
4 answers
280 views

How to safely store application data on shutdown?

I am writing a Windows Service using the Windows Background Worker Service template. Some data that I track needs to persist between application instances (i.e. after shutdown and restart). Initially, ...
PatJon6's user avatar
1 vote
0 answers
87 views

Design patterns for versioning steps across data/workflow pipelines?

I'm sure this has been touched upon by a number of questions, but I'm struggling on drawing the boundaries between code, data and configuration versions when working with a large DAG (think airflow or ...
MYK's user avatar
  • 343
0 votes
2 answers
81 views

How do I control many datasets that are related and can change independently?

Background: I'm a scientist working with a small team on data analytics. No one on our team has any experience with software engineering, though I'm trying to change that. Suppose I have an array A ...
coney1's user avatar
  • 87
-3 votes
2 answers
80 views

Optimal variable-time logging of a real-time data stream

Cross-posted from stats stackexchange Say I have a logging utility in my application that I use for recording timestamped diagnostic log messages. I want to add tracking of some performance metrics to ...
user3716267's user avatar
0 votes
1 answer
69 views

load testing - spike vs stress to determine performance

During the course of my load testing journey, I'm doing mostly spike test on the SUT(System Under Test). We gradually dial up the number of users hitting the server, from 10, 100, 1000. And we log the ...
juztcode's user avatar
  • 139
3 votes
2 answers
195 views

How to propagate seldomly changing data in a distributed system

We store data – some kind of metadata like car manufacturers or types of computer parts – in one of our applications. This data changes rarely. Let's say it changes once every two weeks. This data is ...
Harold L. Brown's user avatar
1 vote
1 answer
365 views

Transform scanned PDF documents into Excel using OCR

I have a task that is to convert cable schedules into an Excel spreadsheet. I have tried a few different OCR (Optical Character Recognition) things such as websites, code in R (tesseract), javascript, ...
Iestyn's user avatar
  • 29
0 votes
1 answer
89 views

How is model reconstructed with event sourcing in CQRS principle?

I'm trying to understand the CQRS design. As far as I know now, every time a change to a model is saved, the (state) change is stored as a separate record. When reading a model, the model is ...
user2190492's user avatar
0 votes
1 answer
99 views

UML design for data analytics of aggregated Survey Data

First and foremost, let me say that I do not have a Software Engineering background . I need help from the community as I have been assigned to create a UML (Unified Modeling Language) design from ...
user3115933's user avatar
1 vote
2 answers
292 views

Whose job actually is to process data in the database to become something like a chart in frontend?

I'm new here, so please be gentle. I've been having issues at my workplace about the front-end team demanding me as a backend to provide every piece of data that has to be displayed in the front end. ...
Alif Irhas's user avatar
1 vote
2 answers
68 views

If I want to add 'period' in the database (one month), and would like to make calculations with the first day of the period. What would I have to do? [closed]

I have a question. I want to add 'period' in the database. A period would always be 1 month. How Would I be able to make calculations with the period? For example I want to calculate the following: A =...
Rquestions's user avatar
5 votes
2 answers
263 views

How did the IEEE 754 standard choose how many bits to allocate to each field (i.e. sign, exponent, mantissa)?

I've been reading about the IEEE 754 standard, but can't understand how or why they chose those exact number of bits for the exponent and mantissa. For example, why 8 bits for the exponent and not 7 ...
hackm's user avatar
  • 53
0 votes
3 answers
286 views

What's stopping an app from using another platform's storage, for free?

Let me clarify on a bit of an abstract question. I'm about to develop an app, through which users can capture and store images. Commercially, the main cost here that springs to mind is storage cost. ...
DanDev's user avatar
  • 23
2 votes
3 answers
197 views

Eliminating combinations based on user input

I am stuck with this particular problem. To give context to the problem, I am developing a mobile app which helps with loading dangerous goods onto a truck. Ignore the size and weight of the dangerous ...
Saul Goodman's user avatar
2 votes
1 answer
218 views

User (Actor) data shared across Domains / Microservices

I have read through a number of posts on Software Engineering but am unable to find a suitable solution for the problem my team and I are trying to solve. We are building a system where we have ...
Richard's user avatar
  • 73
1 vote
0 answers
717 views

Best way to store key-value pairs of different types in c++?

I'm reading values from a (PE) binary file. The values have a known length and position in the file and are stored with no padding (right next to each other). They store various types (int, long, ...
jan's user avatar
  • 111
1 vote
1 answer
113 views

What is a common practice to refer to other datas within a data oriented design?

This question has been in my mind for a while now, especially in the context of high performance, interactive 3d applications. Just want to find out what is the general practices in DoD for ...
Gasim's user avatar
  • 179
1 vote
3 answers
727 views

Repository w/ MVVM Pattern: Data Format Check, Where/When, Best Practice?

TL; DR Where/When to check data format in MVVM + Repository Pattern is the best practice? Any suggestion or theory? The Scene Our team is working on an Android Project. We have a MVVM+repo structure: ...
Samuel T. Chou's user avatar
0 votes
3 answers
164 views

Can you suggest ways to assure client data confidentiality in a SaaS product managed by yourself?

I am designing a multitenant SaaS platform where the database will hold information for different clients. The application consists of an Angular frontend and a REST API Backend. Some of the clients ...
codependent's user avatar
-3 votes
1 answer
182 views

Adding rows and columns simultaneously to pandas dataframe

I am new to Python (~ 1 month) and am having a hard time figuring out a clean, fast way to simultaneously append rows and columns to a dataframe automatically. As a representative example, I will ...
Giovanni Ninivaggi's user avatar
0 votes
2 answers
343 views

Share data between users without a database, php [closed]

How would I create for example a live chat, where you "post" a piece of text and it is displayed to other users in a "chat room", without the need of permanently storing it. My ...
hubbledeepfield's user avatar
-1 votes
1 answer
41 views

Scalable Web Data Service Approach

I'm looking to build a service that interacts with an existing web application's database. The service would retrieve data from an api for which the web application users own an account and warehouse/...
nojoma's user avatar
  • 1
0 votes
1 answer
85 views

Data Class: spread boilerplate for each different database over multiple class copies, or add complexity to original class?

I'm creating a prototype mobile app that will store data locally in an sqLite database and also on the cloud (CloudKit for now). The cloud storage uses a certain way of storing the data, and sqlite ...
Adam B's user avatar
  • 1,575
1 vote
1 answer
155 views

Integration with stripe, have a single source of truth or have info of the user status in our DB

So We're integrating our application with stripe and we have different products There are 2 options : Have stripe as a single source of truth, anytime we need to check the permissions of a user we ...
Dany Y's user avatar
  • 133
1 vote
1 answer
408 views

SQL Foreign Key Relationships and External Data

I work for a company that wants an external system built outside of their core system. I can't share an SQL database, and can't store data in their database. So I am forced to maintain my own SQL ...
Tom Crosman's user avatar
1 vote
1 answer
43 views

Correct/specific term for renaming/restructuring objects

I am developing an app whose main purpose is to display data in tables, lists and graphs. As such, sometimes I need to rename or restructure some of the data recieved from the backend into a "...
MaxPower's user avatar
0 votes
1 answer
118 views

Should you expose API endpoints on an application that is under heavy load or delegate it to another application?

Say I have an application/service A. This application will continuously take in some input X, process it and store it in some database T. This process is quite resource intensive. Now you also wish, ...
rarchim2's user avatar
2 votes
2 answers
1k views

Duplication of data vs loose coupling

Currently, I'm working with a health management application, let's call it application A. It's a partly prebuilt generic application that we are extending on. The main entities are journals, events, ...
XRaycat's user avatar
  • 131
-1 votes
1 answer
395 views

How to optimize the request time for large data response?

I have created a dashboard for rendering a list of clients into a DataTable. Below shown is the data model structure: When I had a few records in the clients schema, let's say a thousand rows, the ...
Saud's user avatar
  • 109
0 votes
2 answers
349 views

Data structure for graphs and vectors in C

I'd like to design an implementation for graphs in C. I'm wondering what the most efficient approach in terms of both computational power and memory consumption would be. I've looked at various open-...
j3141592653589793238's user avatar
-1 votes
1 answer
35 views

Time limit or built-in flush of HashTable entries

Is there a standard data type which is a struct or a table or a dictionary which will always delete objects that are older than t minutes? for example if I want to make sessions expire after 60 ...
Niklas Rosencrantz's user avatar
1 vote
4 answers
338 views

Thinking about data as object vs table row [closed]

It often happens that we have a set of structured data. Let's say our data is about charitable organizations. I could turn it into a list/set of objects/structs: struct Charity { let name: String ...
Eerik Sven Puudist's user avatar
0 votes
3 answers
1k views

Can more than one memory address be accessed at one time?

Apologies if this isn't the correct place to ask the question. But, I'm stumped. Can multiple memory addresses be read/written at one time, or must one address be read then another, is it possible for ...
jdow's user avatar
  • 194
5 votes
2 answers
777 views

What happens to the rest of memory address space if the data is less than 1 byte?

Sorry if this sounds like a stupid question I’m very new to this. So my question is; within modern computers memory address space is a byte long, so what would happen to the rest of the address space ...
jdow's user avatar
  • 194
1 vote
2 answers
92 views

Using objects to construct nearly the same objects

I am getting worried I have some terrible design pattern here in JS/Node.js/MongoDB. It seems to that to create an object, I need an object that looks like that object. This includes data that comes ...
allthewayaround's user avatar
0 votes
0 answers
72 views

How should data be formatted between embedded device and app/cloud?

We have an embedded device collecting data from various sensors attached to it. The data is then passed to the application. What is the best way to design the data packets? The data needs to be ...
Galactasm's user avatar
1 vote
3 answers
296 views

Is it a good practice to isolate the database from public APIs?

We develop web-applications using a javascript frontend framework (ReactJS), to retrieve the data to show, the user's browser calls an authenticated REST API which is our backend (Kotlin+Spring), the ...
molok's user avatar
  • 119
0 votes
1 answer
366 views

Python: Function pipeline with multiple return/input values, or use OOP? Best Practices?

I have a 'processing' function and a 'serializing' function. Currently the processor returns 4 different types of data structures to be serialized in different ways. Looking for the best practise on ...
Jamal Rahman's user avatar
1 vote
0 answers
306 views

How to (simply) architecture a way to ingest multiple types of large files, process them, and send data in chunks to web services?

Note: All of this would be in AWS Hi everyone, What would you guys suggest for building something that: Takes in several different input file types (ex: csv, json, jsonl, xml, .gz, ...) That can be ...
user avatar
5 votes
4 answers
509 views

Static data - database or code

I work on small to medium, database driven, line of business applications. What I usually find, especially in older systems, is most of the data lives in the database. By data I mean stuff like a ...
Uhmmer's user avatar
  • 75
0 votes
2 answers
89 views

Appropriate storage to store all keyboard and mouse events on my computer?

I'd like to build detail analytics of my computer usage to try and detect patterns and improvement opportunities. So I'm building a software to record every mouse click, keyboard event and window ...
Julien__'s user avatar
  • 249
-3 votes
1 answer
2k views

Difference between read and write access (data, IoT)

i am a lawyer and currently writing an essay about data access in ioT environments. I often stumble over the distinction between read access and write access. Could someone provide me with an example, ...
Liam Liam's user avatar
-3 votes
2 answers
2k views

Sending JSON in body of an email

Basically, in our scenario, data has to be sent from a WordPress form to Dynamics 365. My current approach is putting it into an email body in JSON format, that is then extracted and further processed ...
Khadim Ali's user avatar
0 votes
2 answers
127 views

How can I associate algorithm-specific data to the objects it works with, efficiently yet cleanly?

I'm writing in C++, but this problem applies to most non-high-level languages, and possibly some high-level ones as well. I have a graph of heterogeneous nodes. The graph can be instantiated by ...
Helloer's user avatar
  • 253

1
2 3 4 5
9