Data are values of qualitative or quantitative variables, belonging to a set of items. Data in computing (or data processing) are represented in a structure, often tabular (represented by rows and columns), a tree (a set of nodes with parent-children relationship) or a graph structure (a set of ...
0
votes
0answers
29 views
Time series database for low-resolution data
Does it make sense using a time series database, such as opentsdb, for very low-resolution data (hourly, daily) or I should rather use a relational/NoSQL database engine instead?
It's quite possible ...
1
vote
0answers
30 views
separate table for one of the type or java enum?
I want to make budget tracking web-app, tracking all savings and expenses with corresponding dates. Later generating reports, summaries etc..
I came up this design
Frequency Table - does it have to ...
1
vote
2answers
52 views
Distributed training of many small ML models
I have a data science application that involves training tens of thousands of small individual Gaussian models. By "small", I mean that any individual model can easily be trained on one of our worker ...
0
votes
0answers
31 views
Combine n-Linear Series into DataGrid
I'm scratching my head at this, I really need to redesign the schema of an old existing data concept we have in our db. The more I work on it, the more I see why they took an easy way out.
Currently, ...
1
vote
0answers
52 views
How can I offload customer's data needs onto the customer?
I write code for an Engineering application where engineers keep giving me various specs in a form of tabular data - for example, a table with product model names and many various parameters for each ...
1
vote
1answer
100 views
File storage microservice design
Problem overview:
I'm creating a spring application just for learning purposes. I would like to create a microservice just for files, which:
at the beginning would have only two basic endpoints ...
3
votes
3answers
170 views
How to store external numeric IDs - as strings or as integers?
Every now and then we programmers need to communicate with external systems (not under our own control). It's frequent that in such cases we need to keep track of some sort of identifiers that these ...
0
votes
1answer
68 views
Battery level prediction model
I have the idea of develop a model that predicts the battery charge level of my system for now until the following 5 days. The battery is charged using a solar panel. I am writing my code in Python 2....
0
votes
0answers
30 views
Development Process for Building Data Intensive Applications
I have been building production ETL applications for the team I am currently part of and the dev process for any given app has been going like:
Initial requirements and data
Build functional ...
1
vote
1answer
40 views
Which datastructure to use for temporary storage of intermittent and asynchronous sensor data?
Here an outline of the setup:
Input data:
I have several sources for sensor input data (GPS, and a bunch of other sensor data from different sources).
All of those individual inputs consist of a ...
-3
votes
2answers
131 views
data vs business logic
Today my manager said: 'Amir, don't mix the business logic with data in database'.
I surprisingly shocked! I said 'Where?'.
According to wikipedia business logic is:
In computer software, business ...
0
votes
0answers
21 views
How to implement dynamic settings in a web app — db table design?
In my web app I want to be able to use Settings as an admin that I can edit/remove/add via my admin page
I think I'll create a table like this:
settings(id: int, key: string, value: string)
I ...
-1
votes
2answers
72 views
How to structure an application (and its user interface) such that extraneous and/or malformed data can be handled?
I recently applied for a position as a software developer. As a part of the recruitment process, I was given an assignment that involves reading data from a file.
The thing is that the example file ...
2
votes
2answers
148 views
Does Wikipedia Store Every Edit Page? [closed]
I am not sure if this is the best place to ask this question, but I was looking at a Wikipedia article and noticed there were a lot of edits for that article. Since you can view each edited iteration ...
1
vote
1answer
72 views
Are data attributes good identifiers of certain blocks of HTML code?
I have a complex environment which is mostly oriented around / ends up spitting out blocks of HTML, which make up the entire page.
Now, I need a way to "point and click" identify any block on my page ...
0
votes
1answer
216 views
Microservices - sharing data through distributed in-memory cache?
I'm fairly new to both microservices and shared in-memory databases like Redis.
I'm wondering though if it would be a good idea to combine the two to address the challenges of data duplication, ...
0
votes
1answer
54 views
Abstract data communication in modular design
I'm doing the analysis for a software which I've had in mind for a long time. Its purpose is to turn sound into an image, apply graphical transformations on it, and turn it back into sound to hear the ...
1
vote
1answer
83 views
Software layers and data mapping
Before I would ask my question, here is the context: I am working on one part of a relatively big software. In its simplest form there are 3 layers of the software lets call them lower layer, middle ...
1
vote
0answers
45 views
Putting the rowmapper inside the domain class
Normally in java the RowMapper is placed in its own class.
Instead, what about this design, putting the RowMapper inside each domain object, and giving all of them a consistent name:
public class ...
0
votes
1answer
121 views
What is the best way to represent data with inheritance?
In my current work, we need to serialize some data and use it later.
Our data is in this format:
<resources>
<style name="Person">
<item name="name">name</item>
<...
1
vote
2answers
259 views
Where to place POCOs validation - N-layers architecture
I am developing an n-layer app.
One of the layer is the BusinessLayer and consumes a set of POCOs defined in the CoreLayer. Also I have a PresentationLayer (WinForms)
CoreLayer
POCOs (classes with ...
1
vote
2answers
79 views
Design patterns / Names for data consolidation
I'm new to coding and I don't know how what I'm trying to do is called, and I want to find out so I can do more research and not re-invent the wheel. I looked for things such as data consolidation/...
-1
votes
1answer
93 views
Does it make sense to iterate a ranged for loop using constant reference here?
I have the following code, and I was wondering if I'm optimally iterating through my ranged for loop:
struct data_type
{
int a;
int b;
};
int main()
{
// Assume I have initialized a ...
-4
votes
1answer
73 views
What options are there to administrate a database for an online business? [closed]
Say you create a small online business from scratch, a nice Rails or Django app or something to that effect.
With luck, you end up with a database full of customer orders and things.
Now, aside ...
0
votes
0answers
39 views
Data goverance design for Hadoop cluster without apache atlas or cloudera navigator
I am working on a hadoop cluster that needs to implement some form of data governance (for my purposes, say just data lineage/provenance for tracking the history of files in the HDFS). This cluster ...
0
votes
1answer
149 views
How to store large test data sets. Git? [duplicate]
We have a camera application that does measurements on pictures. We have a feature for saving a "demo" containing the pictures/data used, and to replay this for test purposes. We use such data sets ...
-2
votes
2answers
113 views
Are there benefits to having a SQL-like read only database layer over a Blockchain distributed ledger? [closed]
Distributed Blockchain datastores provide inmutability to recorded transactions and only allow to add records accepted by a consensus algorithm in sequence. You could store data on the blockchain or ...
1
vote
1answer
57 views
How to solve double withdrawal attacks on decentralized databases
I need a way to solve double withdrawal attacks in decentralized databases.
For example, a banking infrastructure has multiple servers with their own databases, the advantage is that it lets citizens ...
3
votes
2answers
102 views
Is it acceptable to test based on test output data rather than input data in unit tests?
I'm used to write unit tests with assertions which are based on the input, e.g. (hopefully self-explanatory and let's assume that using random test data is fine)
int a = random();
int b = random();
...
3
votes
1answer
97 views
Should files/data follow rules like 'open-closed principle'?
Does 'open-closed principle' also includes files and data of the app? According to https://softwareengineering.stackexchange.com/a/296955/248528, it says files and data are't included, but it doesn't ...
1
vote
2answers
88 views
Bridging application development practices and game development practices
I previously ran into an interesting practice called occlusion culling in game development. I am still researching to gain more understanding of how it works, since this is mostly discussed in game ...
1
vote
2answers
1k views
Keeping sensitive data encrypted in memory
I'm working in a Electronic Funds Transfer (EFT) system and need be very careful with sensitive data like credit card numbers.
We need this information do mount the ISO 8583 data before sending to ...
-1
votes
2answers
260 views
What is data persistence in the context of software engineering?
I imagine it is simply, as it is named, the existence of data throughout layers of a software application. I ask because I have not been able to find a clear definition that states something of the ...
4
votes
4answers
977 views
Binary data formats, how to make ensure you can read different format versions?
On our project we have this data format that we use to process and record data on. As of late our application changed so that many of the data formats parameters have become obsolete. Currently we ...
-2
votes
1answer
155 views
How does Steam resume downloads without computing a partial hash?
The Steam Client application allows one to download games purchased through the associated Steam account. A typical modern game uses quite a bit of disk space (in the order of 50-70 GB), so it is not ...
-2
votes
2answers
41 views
Security of data viewed and altered by a web app
I am working on a system that needs to keep the data safe and not accessible to people without authorized access. The client is a web app that communicates with the servers via APIs. This is what I ...
1
vote
1answer
101 views
Should repeatedly-computed quantities derived from member data be stored in member data containers?
This question is generally regarding when it is a good idea to store quantities derived from member data in, say, std::vectors that will be repeatedly computed/accessed in various methods, but I will ...
1
vote
2answers
65 views
Import configuration and relational data
Suppose that a user has multiple items. Each item would have complex configuration - multiple tables with relationships.
Each configuration table would have an id and a item_id which is a FK to items ...
1
vote
1answer
60 views
What is the best way to load data from DB upon application launch?
I am writing an iOS application in XCode. The data loaded for the multitudes of viewcontrollers is generally loaded from the DB upon the viewDidLoad() or viewDidAppear() for that particular VC, or it ...
6
votes
5answers
335 views
Why is it considered a good practice to isolate data management in a microservice?
I have seen proponents of isolating data management and storage into microservices in a way that each microservice has its own data store (at least in a logical sense). Hence, joins and aggregations ...
0
votes
1answer
79 views
Serving heterogenous data from backend
There is an application which serves a huge list of resources and in the UI the users are able to filter the resources by type and other properties.
The resources are of different types but they have ...
6
votes
1answer
806 views
Is there any use case for using 'L' letter after a number literal in C?
I know that in C code using L after a number tells the compiler that the respective number is L.
However I do not see any practical use for this. Do modern compilers still have a use for this ...
36
votes
8answers
11k views
Would having 4 states per “bit” rather than 2 mean twice the storage space?
Pretty straightforward fundamental, albeit naive, question:
Would having 4 states per "bit" rather than 2 mean twice the storage space? In case that isn't clear, I mean as in if every "storage ...
7
votes
4answers
566 views
Database design for fields that should be NOT NULL eventually, but may not yet have a value
I have encountered this problem a number of times. Let's say you have a large object that you want to store in the database. Perhaps it's a dating profile for a user, and it has 100 fields. All of ...
1
vote
0answers
132 views
Collecting sports data from many sources in many formats
I'm trying to solve a problem that exists for a sport. The sport is fragmented into many federations, and each federation is completely isolated from other federations. The sport is the same for each ...
0
votes
0answers
311 views
Database schema change tracking
We are currently using MySQL for our Back-end and what we do to track database changes on development that will eventually be applied to our production environment is done manually. By manual I mean ...
0
votes
4answers
2k views
If C++ is generally faster than java, why most data analysis / ETL tools are developed in java? [closed]
I've read in a lot of forums, a lot of users agree C++ is faster than Java (even if it leads for a microsecond, it is important), so why are the majority of data mining software or software tools ...
3
votes
1answer
68 views
Data to pre-populate a database kept in text files
Many web application require a seed file -- a file, written in the same language as a web application, where I have data to pre-populate a database. This makes sense.
Nonetheless, I sometimes see ...
1
vote
1answer
86 views
Catching base Exception to preserve data integrity
I know that often catching all exceptions (C#: catch(Exception exception){...}) is deemed bad practice.
However, I believe that there are situations where it is perfectly reasonable to do it. For ...
0
votes
1answer
22 views
Design for record linking coordinator
I'm a summer intern at a small company tasked with developing a prototype for a (contact) record linking microservice. After some googling I understand that the first step should be cleanup / ...
