Questions tagged [storage]

Storage - refers to questions where data persistence and storage questions and techniques are asked.

Filter by
Sorted by
Tagged with
0 votes
2 answers
162 views

How does storage virtualization help?

I am trying to understand various types of virtualization techniques. The concept of Hardware virtualization/Virtual machine was simple enough. However, I am struggling to understand the concept of ...
Max's user avatar
  • 131
5 votes
3 answers
954 views

Group set of commands as atomic transactions (C++)

We're designing the architecture of an embedded device (esp32). One of the tasks is that the device should connect to the internet and use a preprovisioned redeem code to register itself with our web ...
glades's user avatar
  • 377
0 votes
3 answers
289 views

What's stopping an app from using another platform's storage, for free?

Let me clarify on a bit of an abstract question. I'm about to develop an app, through which users can capture and store images. Commercially, the main cost here that springs to mind is storage cost. ...
DanDev's user avatar
  • 23
4 votes
2 answers
1k views

Best practice for storing third party tools

Currently my company is storing the installers (and in some cases the installed directory copy) for some open source third party tools that our build uses. Were storing these files in our github repo ...
Slack Groverglow's user avatar
-1 votes
2 answers
454 views

Should I store non-editable items lists in a database?

Relational database is mostly used for storing editable user data. But how about storing non-editable lists of information? Possible examples are: GUI-related: main menu items, selectbox options, ...
user14967413's user avatar
-2 votes
2 answers
62 views

When to self-audit complex calculation [closed]

Given an app that manages appointment bookings, I'm separately storing purchases, bookings made, or cancellations (some of them non-refundable) and am required to show the sum of these, based on ...
benomatis's user avatar
  • 105
7 votes
7 answers
8k views

What is the size of the number 65535 in bytes? [closed]

As I got to know there are 256 possible combinations to get for 1 byte. If I understand it correctly, it should mean that you can display any number out of numbers 0-255 and this very number would use ...
treesarecool's user avatar
-1 votes
1 answer
409 views

Best way to store and fetch image bytes?

I have a case in my startup where I need to fetch images in bytes as fast as possible. At the moment I'm storing the images in Azure Storage then I fetch them on the run and cache them into my ...
Faisal's user avatar
  • 107
4 votes
1 answer
2k views

Understanding usage of quadtree to store location data

I was reading this article, which talks about storing location-based information. The author says that while Latitude and Longitude info for the locations can be stored in a SQL database, retrieving ...
Ufder's user avatar
  • 209
-3 votes
1 answer
51 views

Serving content to authenticated users

I'm wanting to store lots of data in cloud storage - like S3. I want some data to only be accessed by authorised users, one way I could do this is to authenticate the user on my service, download the ...
Tobi Akinyemi's user avatar
5 votes
2 answers
852 views

What happens to the rest of memory address space if the data is less than 1 byte?

Sorry if this sounds like a stupid question I’m very new to this. So my question is; within modern computers memory address space is a byte long, so what would happen to the rest of the address space ...
jdow's user avatar
  • 194
0 votes
1 answer
848 views

Why shouldn't I be using SQLITE instead of the INI format in all cases?

Backstory In an effort to store partial and full downloads to file in a clean manner, I had utilized QSettings which stores its key value pairs in the INI format in a single file. https://doc.qt.io/qt-...
Anon's user avatar
  • 3,603
0 votes
1 answer
269 views

Is background long running thread a good choice for disk IO?

I am building a storage engine software that allows concurrent data writing, now I have two different choices here: Method 1. Background Long-Running Thread Multiple user threads write to their own ...
WoooHaaaa's user avatar
  • 101
5 votes
12 answers
3k views

Why do or should programmers save data in text based formats like JSON or XML instead of binary?

I see a lot of reasons to use binary over text-based formats. With binary, I find it a lot easier. I can use fread(data, sizeof(struct DataStruct), 1, fileptr) to read, or fwrite to write data. I ...
Block of Diamond's user avatar
1 vote
0 answers
84 views

Streaming File Storage in Distributed (and Containerized) Systems

I'm trying to implement a system that allows a user to upload files over HTTP, saves the file to object storage as well as any metadata surrounding the file that already exists to a NoSQL database (i....
foxtrotuniform6969's user avatar
9 votes
1 answer
160 views

What is the field/discipline called that deals with the automated management of very large and heterogeneous collections of files?

I work for an academic research lab whose file storage footprint is growing past the point where it can be properly managed "by hand", i.e. without extensive automation. (The number of ...
kjo's user avatar
  • 201
1 vote
1 answer
170 views

Should I deploy redundant SQL database servers in a cloud environment?

As a System Architect I need to deploy a resillient database server in an isolated private cloud. The cloud is OpenStack with Ceph backend and the Database is SQL. Is a single database server VM with ...
Adam Ratcliff's user avatar
1 vote
1 answer
174 views

When keeping all state in an SQL DB, how to handle one-off variables?

I noticed this pattern in my programs: I'm building a system, like the one I'm building for my local hackerspace right now. It's using database as its "brain" - a scraper as a separate ...
d33tah's user avatar
  • 143
-1 votes
1 answer
103 views

Ontology + Relational bases or only ontology to store data in a web app?

I am trying to conceive a wine cellar application for my father and for fun on my free time. I've decided to make a simple MVC with Pixi.js / JQuery and Bootstrap served by Express for the front and a ...
Baguette Eater's user avatar
0 votes
0 answers
56 views

Should unused user-uploaded resources (e.g. images) be deleted from storage?

An example web application X allows its clients to upload various resources, e.g. images. These uploaded resources can also be removed by the user at a later time. My question is this: what are the ...
Avius's user avatar
  • 397
3 votes
3 answers
2k views

When would I use a database over an object-store for large blobs?

I am designing an application where there will be one column that stores a large amount of text. I'm debating whether the text should be stored in the database itself, or whether the database should ...
chris.mclennon's user avatar
2 votes
1 answer
78 views

Storing and filtering spatial data within RAM

I'm trying to figure out what's the proper way of storing and filtering spatial data inside a running program. I have a 2D map of theoretically infinite size. Users generate objects which are placed ...
php_nub_qq's user avatar
  • 2,224
-1 votes
2 answers
78 views

Sensitive data and cloud storage

I've been reading about machine learning models deployment and pipelines and so far most of the sources suggest that the data should be ingested from some sort of cloud based storage or source, be it ...
Alex T's user avatar
  • 161
0 votes
1 answer
158 views

Storing persistent settings for each product in list

Sorry if I'm not clear, I didn't touch the program for almost two weeks. I have a program that reads XML files: A product list Each product's default settings (if custom is not available) Each ...
xtl's user avatar
  • 9
1 vote
1 answer
2k views

How do I store a trie in memory in a Spring Boot Application?

I have a spring boot application, that simply does, takes a set of characters and lists out possible english words. Now, as everyone knows , its fairly easy, build a trie data-structure to load up a ...
MithunS's user avatar
  • 127
4 votes
1 answer
8k views

Storage of global variables

In C, global variables are stored with the program code. I.e. the space to hold them is part of the object file (either in the data or bss section), instead of being allocated during execution (to ...
Jet Blue's user avatar
  • 159
2 votes
1 answer
859 views

Azure Key Vault vs Automation Account Variables for password storage

In Microsoft Azure one can store secret values like passwords in Azure Key Vault, but also as encrypted variables in Azure Automation (these ones are stored in an Azure-managed Key Vault behind the ...
Woody Chan's user avatar
0 votes
1 answer
690 views

Recommended Long Term Data Storage Format

I have an application (written in Go) with a lot of GPS location information. Each record consists of a latitude and longitude plus somewhere around 50 fields of various pieces of meta data, some of ...
Brad Peabody's user avatar
1 vote
0 answers
300 views

Is it better to have a default avatar library on client or server side?

I have a webapp using ReactJS as the frontend and Rails as the backend service. I have an account manager, where a user can choose multiple avatars from a default library. This library contains ...
KawaLo's user avatar
  • 129
1 vote
1 answer
133 views

How do database storage engines handle new columns?

I really don't know which StackExchange site this belongs on. It didn't seem to match the "on-topic" list of DBA.SE or SO.SE ... so please let me know if I should move it. I also couldn't find a ...
Simon Whitehead's user avatar
2 votes
2 answers
3k views

Architecture for uploading large files from many end points to the cloud storage

I am working on a desktop app that offers uploading to the cloud storage. Storage providers have an easy way to upload files. You get accessKeyId and secretAccessKey and you are ready to upload. I am ...
sanjihan's user avatar
  • 123
6 votes
1 answer
301 views

How can I reduce the amount of storage needed for a gravitational n-body simulation?

I am currently attempting to create a gravitational n-body simulation using a modified Barnes-Hut algorithm, to be more amicable to GPU computation. This is primarily as a learning project. My goal is ...
john01dav's user avatar
  • 889
-4 votes
2 answers
1k views

How the value of the integer variable is stored in a memory by bytes/bits?

In C program I'm doing below stuff int x = 4; Let us assume integer has 2 bytes in this case. So my question here is the variable x will hold two bytes that mean 16 bits. So here how the value 4 ...
Abdulvakaf K's user avatar
-3 votes
3 answers
420 views

Is storing configurations in memory more performance and maintenance efficient? [closed]

My teammate has this class which contains a lot of Strings: public class Config { /** * List of status */ public static final String IN = "bla*"; public static final String ...
Rigo Sarmiento's user avatar
11 votes
1 answer
741 views

Concurrent fault-safe data structure

I am building an application that aims to process ~10M data items per second. Each item is exactly 42 bytes small (including a sorting key) which means the total data rate will not be big 420 MB/s. ...
SmartArray's user avatar
1 vote
2 answers
9k views

Should a SVG be converted to JSON to be stored in the database?

Consider an application where you can create and edit a SVG file (e.g. draw.io). How should the SVG be stored in the database? With SVG (image/svg+xml) Thus no further processing is required before ...
luukvhoudt's user avatar
1 vote
0 answers
65 views

How can I offload customer's data needs onto the customer?

I write code for an Engineering application where engineers keep giving me various specs in a form of tabular data - for example, a table with product model names and many various parameters for each ...
Dennis's user avatar
  • 8,227
3 votes
3 answers
750 views

Data storage for small application using XML

I am developing a small application as a hobbyist. This application will track costs (eg. purchases and sales, ...) for a single person company. Only string, date, double, integer, and decimal types ...
Mech_Engineer's user avatar
4 votes
3 answers
2k views

How to store external numeric IDs - as strings or as integers?

Every now and then we programmers need to communicate with external systems (not under our own control). It's frequent that in such cases we need to keep track of some sort of identifiers that these ...
Vilx-'s user avatar
  • 5,320
-1 votes
1 answer
51 views

Data store for sequential key-value, time limited event "journal"

I'm designing a system in which a key component acts like a "journal", in the sense that it stores a list of sequential events that can change the state of the entire system. Upon request, any other ...
shevron's user avatar
  • 591
4 votes
2 answers
313 views

Does Wikipedia Store Every Edit Page? [closed]

I am not sure if this is the best place to ask this question, but I was looking at a Wikipedia article and noticed there were a lot of edits for that article. Since you can view each edited iteration ...
user10135146's user avatar
-5 votes
1 answer
177 views

Is this a legit question? [closed]

I see an interview question thats goes like this: Assuming you have an array with numbers 1 to N. Where N is at most 32,000. This array may have duplicates and you do not know what N is. With, 4KB ...
Eugene Lim's user avatar
-2 votes
1 answer
120 views

What is the preferable date-format used for designing APIs and storage

The date format should support different timezone and Daylight saving issues.
Sharath's user avatar
  • 123
2 votes
2 answers
155 views

What is the best practice for pairing objects of a sorted vector?

I have a vector that stores multiple instances of a type T: std::vector<T> vec; From time to time, vec is being sorted for different purposes. Now, I need a list of pairs of objects stored in ...
BalazsToth's user avatar
-7 votes
1 answer
416 views

Importance of storage in c++ [closed]

I want to know that whether storage is an issue with respect to current hardware and software market when we talk about the large size of C++ programs?? Because in certain conditions we have to choose ...
Tom 's user avatar
  • 1
4 votes
4 answers
3k views

Binary data formats, how to make ensure you can read different format versions?

On our project we have this data format that we use to process and record data on. As of late our application changed so that many of the data formats parameters have become obsolete. Currently we ...
Krupip's user avatar
  • 1,300
0 votes
1 answer
2k views

What is the recommended way to storage documents as Microservice / API?

I'am new in Microservices, and this is the scenario. Background: I have an Web Application that requires store and retrieve files and let's say that i need to storage documents (PDFs, Word Documents,...
Alfred Severo's user avatar
-3 votes
1 answer
69 views

De-duplication procedure [closed]

I knew that by using De-duplication procedure we can save storage costs. 1) I know what is de-duplication but I don't know how it works. Can someone clearly explain how de-duplication procedure works?...
SRIDHARAN's user avatar
  • 125
35 votes
8 answers
13k views

Would having 4 states per "bit" rather than 2 mean twice the storage space?

Pretty straightforward fundamental, albeit naive, question: Would having 4 states per "bit" rather than 2 mean twice the storage space? In case that isn't clear, I mean as in if every "storage ...
J.Todd's user avatar
  • 3,823
13 votes
3 answers
8k views

How are variables stored in a language compiler or interpreter?

Say we set a variable in Python. five = 5 Boom. What I'm wondering is, how is this stored? Does the compiler or interpreter just put it in a variable like so? varname = ["five"] varval = [5] If ...
baranskistad's user avatar