Questions tagged [storage]

Storage - refers to questions where data persistence and storage questions and techniques are asked.

Filter by
Sorted by
Tagged with
3 votes
2 answers
135 views

Best practice for storing third party tools

Currently my company is storing the installers (and in some cases the installed directory copy) for some open source third party tools that our build uses. Were storing these files in our github repo ...
user avatar
-1 votes
2 answers
142 views

Should I store non-editable items lists in a database?

Relational database is mostly used for storing editable user data. But how about storing non-editable lists of information? Possible examples are: GUI-related: main menu items, selectbox options, ...
user avatar
-2 votes
2 answers
58 views

When to self-audit complex calculation [closed]

Given an app that manages appointment bookings, I'm separately storing purchases, bookings made, or cancellations (some of them non-refundable) and am required to show the sum of these, based on ...
user avatar
  • 105
8 votes
7 answers
5k views

What is the size of the number 65535 in bytes? [closed]

As I got to know there are 256 possible combinations to get for 1 byte. If I understand it correctly, it should mean that you can display any number out of numbers 0-255 and this very number would use ...
user avatar
-1 votes
1 answer
97 views

Best way to store and fetch image bytes?

I have a case in my startup where I need to fetch images in bytes as fast as possible. At the moment I'm storing the images in Azure Storage then I fetch them on the run and cache them into my ...
user avatar
  • 107
1 vote
1 answer
509 views

Understanding usage of quadtree to store location data

I was reading this article, which talks about storing location-based information. The author says that while Latitude and Longitude info for the locations can be stored in a SQL database, retrieving ...
user avatar
  • 169
-3 votes
1 answer
43 views

Serving content to authenticated users

I'm wanting to store lots of data in cloud storage - like S3. I want some data to only be accessed by authorised users, one way I could do this is to authenticate the user on my service, download the ...
user avatar
5 votes
2 answers
472 views

What happens to the rest of memory address space if the data is less than 1 byte?

Sorry if this sounds like a stupid question I’m very new to this. So my question is; within modern computers memory address space is a byte long, so what would happen to the rest of the address space ...
user avatar
  • 184
-1 votes
1 answer
412 views

Why shouldn't I be using SQLITE instead of the INI format in all cases?

Backstory In an effort to store partial and full downloads to file in a clean manner, I had utilized QSettings which stores its key value pairs in the INI format in a single file. https://doc.qt.io/qt-...
user avatar
  • 3,319
0 votes
1 answer
68 views

Is background long running thread a good choice for disk IO?

I am building a storage engine software that allows concurrent data writing, now I have two different choices here: Method 1. Background Long-Running Thread Multiple user threads write to their own ...
user avatar
  • 101
5 votes
12 answers
1k views

Why do or should programmers save data in text based formats like JSON or XML instead of binary?

I see a lot of reasons to use binary over text-based formats. With binary, I find it a lot easier. I can use fread(data, sizeof(struct DataStruct), 1, fileptr) to read, or fwrite to write data. I ...
user avatar
1 vote
0 answers
50 views

Streaming File Storage in Distributed (and Containerized) Systems

I'm trying to implement a system that allows a user to upload files over HTTP, saves the file to object storage as well as any metadata surrounding the file that already exists to a NoSQL database (i....
user avatar
8 votes
1 answer
146 views

What is the field/discipline called that deals with the automated management of very large and heterogeneous collections of files?

I work for an academic research lab whose file storage footprint is growing past the point where it can be properly managed "by hand", i.e. without extensive automation. (The number of ...
user avatar
  • 191
1 vote
1 answer
120 views

Should I deploy redundant SQL database servers in a cloud environment?

As a System Architect I need to deploy a resillient database server in an isolated private cloud. The cloud is OpenStack with Ceph backend and the Database is SQL. Is a single database server VM with ...
user avatar
1 vote
1 answer
92 views

When keeping all state in an SQL DB, how to handle one-off variables?

I noticed this pattern in my programs: I'm building a system, like the one I'm building for my local hackerspace right now. It's using database as its "brain" - a scraper as a separate ...
user avatar
  • 143
-1 votes
1 answer
66 views

Ontology + Relational bases or only ontology to store data in a web app?

I am trying to conceive a wine cellar application for my father and for fun on my free time. I've decided to make a simple MVC with Pixi.js / JQuery and Bootstrap served by Express for the front and a ...
user avatar
0 votes
0 answers
52 views

Should unused user-uploaded resources (e.g. images) be deleted from storage?

An example web application X allows its clients to upload various resources, e.g. images. These uploaded resources can also be removed by the user at a later time. My question is this: what are the ...
user avatar
  • 357
2 votes
3 answers
694 views

When would I use a database over an object-store for large blobs?

I am designing an application where there will be one column that stores a large amount of text. I'm debating whether the text should be stored in the database itself, or whether the database should ...
user avatar
2 votes
1 answer
71 views

Storing and filtering spatial data within RAM

I'm trying to figure out what's the proper way of storing and filtering spatial data inside a running program. I have a 2D map of theoretically infinite size. Users generate objects which are placed ...
user avatar
  • 2,154
-1 votes
2 answers
65 views

Sensitive data and cloud storage

I've been reading about machine learning models deployment and pipelines and so far most of the sources suggest that the data should be ingested from some sort of cloud based storage or source, be it ...
user avatar
  • 151
0 votes
1 answer
119 views

Storing persistent settings for each product in list

Sorry if I'm not clear, I didn't touch the program for almost two weeks. I have a program that reads XML files: A product list Each product's default settings (if custom is not available) Each ...
user avatar
  • 9
1 vote
1 answer
1k views

How do I store a trie in memory in a Spring Boot Application?

I have a spring boot application, that simply does, takes a set of characters and lists out possible english words. Now, as everyone knows , its fairly easy, build a trie data-structure to load up a ...
user avatar
  • 127
4 votes
1 answer
5k views

Storage of global variables

In C, global variables are stored with the program code. I.e. the space to hold them is part of the object file (either in the data or bss section), instead of being allocated during execution (to ...
user avatar
  • 159
1 vote
1 answer
542 views

Azure Key Vault vs Automation Account Variables for password storage

In Microsoft Azure one can store secret values like passwords in Azure Key Vault, but also as encrypted variables in Azure Automation (these ones are stored in an Azure-managed Key Vault behind the ...
user avatar
0 votes
1 answer
392 views

Recommended Long Term Data Storage Format

I have an application (written in Go) with a lot of GPS location information. Each record consists of a latitude and longitude plus somewhere around 50 fields of various pieces of meta data, some of ...
user avatar
1 vote
0 answers
157 views

Is it better to have a default avatar library on client or server side?

I have a webapp using ReactJS as the frontend and Rails as the backend service. I have an account manager, where a user can choose multiple avatars from a default library. This library contains ...
user avatar
  • 129
0 votes
1 answer
76 views

How do database storage engines handle new columns?

I really don't know which StackExchange site this belongs on. It didn't seem to match the "on-topic" list of DBA.SE or SO.SE ... so please let me know if I should move it. I also couldn't find a ...
user avatar
2 votes
2 answers
2k views

Architecture for uploading large files from many end points to the cloud storage

I am working on a desktop app that offers uploading to the cloud storage. Storage providers have an easy way to upload files. You get accessKeyId and secretAccessKey and you are ready to upload. I am ...
user avatar
  • 123
6 votes
1 answer
252 views

How can I reduce the amount of storage needed for a gravitational n-body simulation?

I am currently attempting to create a gravitational n-body simulation using a modified Barnes-Hut algorithm, to be more amicable to GPU computation. This is primarily as a learning project. My goal is ...
user avatar
  • 879
-4 votes
2 answers
1k views

How the value of the integer variable is stored in a memory by bytes/bits?

In C program I'm doing below stuff int x = 4; Let us assume integer has 2 bytes in this case. So my question here is the variable x will hold two bytes that mean 16 bits. So here how the value 4 ...
user avatar
-3 votes
3 answers
278 views

Is storing configurations in memory more performance and maintenance efficient? [closed]

My teammate has this class which contains a lot of Strings: public class Config { /** * List of status */ public static final String IN = "bla*"; public static final String ...
user avatar
11 votes
1 answer
661 views

Concurrent fault-safe data structure

I am building an application that aims to process ~10M data items per second. Each item is exactly 42 bytes small (including a sorting key) which means the total data rate will not be big 420 MB/s. ...
user avatar
1 vote
2 answers
7k views

Should a SVG be converted to JSON to be stored in the database?

Consider an application where you can create and edit a SVG file (e.g. draw.io). How should the SVG be stored in the database? With SVG (image/svg+xml) Thus no further processing is required before ...
user avatar
1 vote
0 answers
60 views

How can I offload customer's data needs onto the customer?

I write code for an Engineering application where engineers keep giving me various specs in a form of tabular data - for example, a table with product model names and many various parameters for each ...
user avatar
  • 8,031
3 votes
3 answers
576 views

Data storage for small application using XML

I am developing a small application as a hobbyist. This application will track costs (eg. purchases and sales, ...) for a single person company. Only string, date, double, integer, and decimal types ...
user avatar
3 votes
3 answers
1k views

How to store external numeric IDs - as strings or as integers?

Every now and then we programmers need to communicate with external systems (not under our own control). It's frequent that in such cases we need to keep track of some sort of identifiers that these ...
user avatar
  • 5,134
-1 votes
1 answer
27 views

Data store for sequential key-value, time limited event "journal"

I'm designing a system in which a key component acts like a "journal", in the sense that it stores a list of sequential events that can change the state of the entire system. Upon request, any other ...
user avatar
  • 531
4 votes
2 answers
263 views

Does Wikipedia Store Every Edit Page? [closed]

I am not sure if this is the best place to ask this question, but I was looking at a Wikipedia article and noticed there were a lot of edits for that article. Since you can view each edited iteration ...
user avatar
-5 votes
1 answer
166 views

Is this a legit question? [closed]

I see an interview question thats goes like this: Assuming you have an array with numbers 1 to N. Where N is at most 32,000. This array may have duplicates and you do not know what N is. With, 4KB ...
user avatar
-2 votes
1 answer
118 views

What is the preferable date-format used for designing APIs and storage

The date format should support different timezone and Daylight saving issues.
user avatar
  • 123
2 votes
2 answers
148 views

What is the best practice for pairing objects of a sorted vector?

I have a vector that stores multiple instances of a type T: std::vector<T> vec; From time to time, vec is being sorted for different purposes. Now, I need a list of pairs of objects stored in ...
user avatar
-8 votes
1 answer
379 views

Importance of storage in c++ [closed]

I want to know that whether storage is an issue with respect to current hardware and software market when we talk about the large size of C++ programs?? Because in certain conditions we have to choose ...
user avatar
  • 1
4 votes
4 answers
2k views

Binary data formats, how to make ensure you can read different format versions?

On our project we have this data format that we use to process and record data on. As of late our application changed so that many of the data formats parameters have become obsolete. Currently we ...
user avatar
  • 948
0 votes
1 answer
2k views

What is the recommended way to storage documents as Microservice / API?

I'am new in Microservices, and this is the scenario. Background: I have an Web Application that requires store and retrieve files and let's say that i need to storage documents (PDFs, Word Documents,...
user avatar
-3 votes
1 answer
68 views

De-duplication procedure [closed]

I knew that by using De-duplication procedure we can save storage costs. 1) I know what is de-duplication but I don't know how it works. Can someone clearly explain how de-duplication procedure works?...
user avatar
  • 125
35 votes
8 answers
13k views

Would having 4 states per "bit" rather than 2 mean twice the storage space?

Pretty straightforward fundamental, albeit naive, question: Would having 4 states per "bit" rather than 2 mean twice the storage space? In case that isn't clear, I mean as in if every "storage ...
user avatar
  • 3,843
12 votes
3 answers
6k views

How are variables stored in a language compiler or interpreter?

Say we set a variable in Python. five = 5 Boom. What I'm wondering is, how is this stored? Does the compiler or interpreter just put it in a variable like so? varname = ["five"] varval = [5] If ...
user avatar
7 votes
5 answers
15k views

Is it ineffecient to store data as json in plain text? [duplicate]

We needed a persistent storage for API keys and I came up with reading and writing json in plain text and the user thinks it works but is it very ineffecient compared to a relational dbms? I just ...
user avatar
35 votes
5 answers
19k views

Does storing plain text data take up less space than storing the equivalent message in binary?

As a web developer I have very little understanding of binary data. If I take the sentence "Hello world.", convert it to binary, and store it as binary in an SQL database, it seems like the 1s and ...
user avatar
  • 977
1 vote
2 answers
398 views

How do I best store a value with multiple keys?

I need to store two 32-bit numbers in a way that is quickly accessible and as storage space efficient as possible. My keys are a combination of the following two values: port: 0 - 29 vlan: 1 - 4095 ...
user avatar
  • 139