Questions tagged [storage]

Storage - refers to questions where data persistence and storage questions and techniques are asked.

Filter by
Sorted by
Tagged with
0
votes
0answers
46 views

Should unused user-uploaded resources (e.g. images) be deleted from storage?

An example web application X allows its clients to upload various resources, e.g. images. These uploaded resources can also be removed by the user at a later time. My question is this: what are the ...
2
votes
3answers
196 views

When would I use a database over an object-store for large blobs?

I am designing an application where there will be one column that stores a large amount of text. I'm debating whether the text should be stored in the database itself, or whether the database should ...
2
votes
1answer
61 views

Storing and filtering spatial data within RAM

I'm trying to figure out what's the proper way of storing and filtering spatial data inside a running program. I have a 2D map of theoretically infinite size. Users generate objects which are placed ...
-1
votes
2answers
52 views

Sensitive data and cloud storage

I've been reading about machine learning models deployment and pipelines and so far most of the sources suggest that the data should be ingested from some sort of cloud based storage or source, be it ...
0
votes
1answer
104 views

Storing persistent settings for each product in list

Sorry if I'm not clear, I didn't touch the program for almost two weeks. I have a program that reads XML files: A product list Each product's default settings (if custom is not available) Each ...
0
votes
1answer
73 views

How do I store a trie in memory in a Spring Boot Application?

I have a spring boot application, that simply does, takes a set of characters and lists out possible english words. Now, as everyone knows , its fairly easy, build a trie data-structure to load up a ...
1
vote
1answer
113 views

Storage of global variables

In C, global variables are stored with the program code. I.e. the space to hold them is part of the object file (either in the data or bss section), instead of being allocated during execution (to ...
0
votes
1answer
78 views

Azure Key Vault vs Automation Account Variables for password storage

In Microsoft Azure one can store secret values like passwords in Azure Key Vault, but also as encrypted variables in Azure Automation (these ones are stored in an Azure-managed Key Vault behind the ...
0
votes
1answer
92 views

Recommended Long Term Data Storage Format

I have an application (written in Go) with a lot of GPS location information. Each record consists of a latitude and longitude plus somewhere around 50 fields of various pieces of meta data, some of ...
0
votes
0answers
43 views

Is it better to have a default avatar library on client or server side?

I have a webapp using ReactJS as the frontend and Rails as the backend service. I have an account manager, where a user can choose multiple avatars from a default library. This library contains ...
0
votes
1answer
46 views

How do database storage engines handle new columns?

I really don't know which StackExchange site this belongs on. It didn't seem to match the "on-topic" list of DBA.SE or SO.SE ... so please let me know if I should move it. I also couldn't find a ...
2
votes
2answers
218 views

Architecture for uploading large files from many end points to the cloud storage

I am working on a desktop app that offers uploading to the cloud storage. Storage providers have an easy way to upload files. You get accessKeyId and secretAccessKey and you are ready to upload. I am ...
6
votes
1answer
238 views

How can I reduce the amount of storage needed for a gravitational n-body simulation?

I am currently attempting to create a gravitational n-body simulation using a modified Barnes-Hut algorithm, to be more amicable to GPU computation. This is primarily as a learning project. My goal is ...
-4
votes
2answers
521 views

How the value of the integer variable is stored in a memory by bytes/bits?

In C program I'm doing below stuff int x = 4; Let us assume integer has 2 bytes in this case. So my question here is the variable x will hold two bytes that mean 16 bits. So here how the value 4 ...
-3
votes
3answers
156 views

Is storing configurations in memory more performance and maintenance efficient? [closed]

My teammate has this class which contains a lot of Strings: public class Config { /** * List of status */ public static final String IN = "bla*"; public static final String ...
7
votes
0answers
358 views

Concurrent fault-safe data structure

I am building an application that aims to process ~10M data items per second. Each item is exactly 42 bytes small (including a sorting key) which means the total data rate will not be big 420 MB/s. ...
0
votes
2answers
2k views

Should a SVG be converted to JSON to be stored in the database?

Consider an application where you can create and edit a SVG file (e.g. draw.io). How should the SVG be stored in the database? With SVG (image/svg+xml) Thus no further processing is required before ...
1
vote
0answers
58 views

How can I offload customer's data needs onto the customer?

I write code for an Engineering application where engineers keep giving me various specs in a form of tabular data - for example, a table with product model names and many various parameters for each ...
3
votes
3answers
166 views

Data storage for small application using XML

I am developing a small application as a hobbyist. This application will track costs (eg. purchases and sales, ...) for a single person company. Only string, date, double, integer, and decimal types ...
3
votes
3answers
289 views

How to store external numeric IDs - as strings or as integers?

Every now and then we programmers need to communicate with external systems (not under our own control). It's frequent that in such cases we need to keep track of some sort of identifiers that these ...
-1
votes
1answer
22 views

Data store for sequential key-value, time limited event “journal”

I'm designing a system in which a key component acts like a "journal", in the sense that it stores a list of sequential events that can change the state of the entire system. Upon request, any other ...
3
votes
2answers
162 views

Does Wikipedia Store Every Edit Page? [closed]

I am not sure if this is the best place to ask this question, but I was looking at a Wikipedia article and noticed there were a lot of edits for that article. Since you can view each edited iteration ...
-5
votes
1answer
146 views

Is this a legit question? [closed]

I see an interview question thats goes like this: Assuming you have an array with numbers 1 to N. Where N is at most 32,000. This array may have duplicates and you do not know what N is. With, 4KB ...
-2
votes
1answer
117 views

What is the preferable date-format used for designing APIs and storage

The date format should support different timezone and Daylight saving issues.
2
votes
2answers
114 views

What is the best practice for pairing objects of a sorted vector?

I have a vector that stores multiple instances of a type T: std::vector<T> vec; From time to time, vec is being sorted for different purposes. Now, I need a list of pairs of objects stored in ...
-8
votes
1answer
340 views

Importance of storage in c++ [closed]

I want to know that whether storage is an issue with respect to current hardware and software market when we talk about the large size of C++ programs?? Because in certain conditions we have to choose ...
4
votes
4answers
2k views

Binary data formats, how to make ensure you can read different format versions?

On our project we have this data format that we use to process and record data on. As of late our application changed so that many of the data formats parameters have become obsolete. Currently we ...
0
votes
1answer
771 views

What is the recommended way to storage documents as Microservice / API?

I'am new in Microservices, and this is the scenario. Background: I have an Web Application that requires store and retrieve files and let's say that i need to storage documents (PDFs, Word Documents,...
-3
votes
1answer
59 views

De-duplication procedure [closed]

I knew that by using De-duplication procedure we can save storage costs. 1) I know what is de-duplication but I don't know how it works. Can someone clearly explain how de-duplication procedure works?...
35
votes
8answers
12k views

Would having 4 states per “bit” rather than 2 mean twice the storage space?

Pretty straightforward fundamental, albeit naive, question: Would having 4 states per "bit" rather than 2 mean twice the storage space? In case that isn't clear, I mean as in if every "storage ...
8
votes
3answers
3k views

How are variables stored in a language compiler or interpreter?

Say we set a variable in Python. five = 5 Boom. What I'm wondering is, how is this stored? Does the compiler or interpreter just put it in a variable like so? varname = ["five"] varval = [5] If ...
5
votes
5answers
10k views

Is it ineffecient to store data as json in plain text? [duplicate]

We needed a persistent storage for API keys and I came up with reading and writing json in plain text and the user thinks it works but is it very ineffecient compared to a relational dbms? I just ...
32
votes
3answers
13k views

Does storing plain text data take up less space than storing the equivalent message in binary?

As a web developer I have very little understanding of binary data. If I take the sentence "Hello world.", convert it to binary, and store it as binary in an SQL database, it seems like the 1s and ...
1
vote
2answers
271 views

How do I best store a value with multiple keys?

I need to store two 32-bit numbers in a way that is quickly accessible and as storage space efficient as possible. My keys are a combination of the following two values: port: 0 - 29 vlan: 1 - 4095 ...
2
votes
1answer
147 views

Optimal format for storing key/value data optimized for quick key lookup

For a completely fun project, I want to write a Markov chain chat bot. The algorithm used is quite simple - break down incoming sentences into tokens, storing what words tend to come after each token ...
-1
votes
1answer
130 views

Complete statelessness, how to save password on client in secure way

Thinking about statelessness I wonder how I can overcome the issue of saving the user's password on the client side in a secure but also comfy way. I assume the client always sends credentials to the ...
1
vote
1answer
736 views

From web database (PostgreSQL) to mobile (SQLite)

My company employs people who add and edit records in a PostgreSQL database using a web interface. These updates then need to be compiled into a mobile app (Android, iOS) via SQLite and released as a ...
5
votes
5answers
503 views

Is saving disk space a valid reason to forgo migrating to a standard text format (e.g. JSON)?

A while ago I asked a question about custom text data formats, instead of using existing tools such as XML, JSON, YAML, etc. Now, in favor of converting our custom format to a relational database and ...
1
vote
0answers
74 views

Software Defined Storage (SDS): How to choose right moment to lock it for backing up?

I'm developing a server which provides a virtual block device via SCSI interface. The device is constantly bombarded by requests to read/write data blocks. Sometimes I need to lock this device (or ...
1
vote
1answer
10k views

Is cloud storage such as Dropbox SaaS, PaaS, IaaS or something else entirely? [closed]

I am stuck at how to classify Dropbox and other cloud storage services within the ISP (IaaS,PaaS,SaaS) model. At their core, such services provide a cloud-backed file system which can be read and ...
0
votes
2answers
229 views

Should MySQL database tables only have a single unit of information per row?

I run an automated data collection service at work, which records measurements (temperatures, voltages, etc) from about a dozen sensors at a rate of about once a second. The MySQL database table looks ...
82
votes
14answers
20k views

Why do most log files use plain text rather than a binary format?

Logging is something that is necessary but is (relatively) rarely used. As such it can be made much more compact in terms of storage. For example the data most commonly logged like ip, date, time and ...
2
votes
1answer
272 views

Storing/hosting sensitive business data

In order for users of my software to receive auto-updates, their copy of the software will need remote access to ~100mb of private files (manifest, jars) which means I need a server. Currently I don't ...
-2
votes
3answers
531 views

Upload file to S3 or save it locally?

I have a web application that allows users to upload files. My application does processing on these files (parsing, modifying metadata, etc). What approach should I follow? Should I save the files ...
0
votes
1answer
88 views

Design question to best option for storing data in file or DB

I am working on a feature where I need to store some data and I am not sure what should be my best option to store this data. I am sending data to some third party and want to show some part of that ...
1
vote
1answer
148 views

What is a simple way to store data for word count purposes?

I am writing a program to crawl the web and find frequently used words. What would be a good way to store the data? I was thinning of using an RDBMS with a single table (with columns 'word' and 'count'...
3
votes
2answers
336 views

Storing large binary data for integration tests

I have the following requirement for an application: Store large files (order of megabytes) which I will use as input for non regression test and some expected properties of the processing result ...
2
votes
2answers
120 views

When do I prefer a local DB over serialization for local data storage?

I recently started developing a small calendar application, in which I can add and view several calendar entries, which contain several data like name, date, alarm settings and so on. Those values ...
2
votes
1answer
121 views

Is it better to pull all data immediately or as needed?

I have a program that needs information from a database but it doesn't need all of the information at once, although it WILL need it at some point. Would it be better to pull all of the information ...
1
vote
1answer
984 views

How to store data from firebase to analyze at some later date [closed]

I am currently using Firebase with my iOS app. I use it to authenticate users and then allow user interactivity during the life of the app session. I would like to be able to save a snapshot of my ...