Questions tagged [database]

This tag is for general database questions. If you question is specific to SQL or NoSQL, use the corresponding tags instead.

Filter by
Sorted by
Tagged with
-1 votes
0 answers
22 views

What storage solution to use for iterating over data slices?

My main objective is for the solution to be hassle free. Specifically it should be more convenient than using a local file system and manually managing files. I thought of SQLite but I don't see an ...
user avatar
0 votes
3 answers
136 views

Writing a growing simulation that scales

Or ways to get better run time than a nested loop? How would you design this? I hope this is the place to ask semi-vague software engineering questions. My simple simulation is like this: There is a ...
user avatar
  • 119
1 vote
1 answer
100 views

Database design for a reccomendation system

I am working on a project which will have a recommendation system and I am in the database design phase, I have a good foundation in programming but databases is a new topic for me, I have designed a ...
user avatar
  • 11
-3 votes
0 answers
26 views

Structuring a database for an investment portfolio?

would really appreciate some general advice on the structure of an app i'm building. I'm essentially trying to re-create an investment portfolio app using a MERN stack. Something very similar to ...
user avatar
-1 votes
2 answers
112 views

Should I store non-editable items lists in a database?

Relational database is mostly used for storing editable user data. But how about storing non-editable lists of information? Possible examples are: GUI-related: main menu items, selectbox options, ...
user avatar
1 vote
2 answers
109 views

Making data available for multiple microservices

We have a design challenge here for a project we are working on and I wonder if folks from the community can provide some guidance: Our product is built in a microservices structure. So we have ...
user avatar
  • 111
-2 votes
0 answers
45 views

How do large storage cloud solutions store their data? [closed]

I was wondering how file upload services / cloud storage solutions actually store their data. Multiple customers from different companies using the same website (E.g. Facebook) and uploading images, ...
user avatar
0 votes
2 answers
75 views

Is it a good idea to use database sequential id as a key in external key-value storage?

I use Postgres database to store user information. Some of that information is sensitive so it was decided to store that information in Vault. The user table in Postgres has id's generated by a ...
user avatar
  • 145
1 vote
0 answers
31 views

Backend Design: System requires both ElasticSearch and MySQL. In which scenario we fetch data from MySQL or ElasticSearch?

So, one of the backend system, having 5 resource types (tables) and some foreign keys as well, requires full text support for two of the tables. Also, for some use-cases, we require inner-joins for ...
user avatar
0 votes
3 answers
118 views

Can you suggest ways to assure client data confidentiality in a SaaS product managed by yourself?

I am designing a multitenant SaaS platform where the database will hold information for different clients. The application consists of an Angular frontend and a REST API Backend. Some of the clients ...
user avatar
1 vote
1 answer
77 views

Contextual Implementation Overhead vs Redundancy

I have an application that has use cases relative to where it is used. A local client and server, where all the server data is in a single location and the client queries that server knowing it is ...
user avatar
16 votes
5 answers
4k views

Why would an "articles" table (with no relationship to any other table) have a primary key?

Let's say that I have an articles table that have two columns: title and content, and let's say that this articles table doesn't have any relationship with any other table. Why would this articles ...
user avatar
0 votes
1 answer
58 views

Synchronous replication of databases

Say we have a software system with a web server serving writes to a Master DB. This Master DB is then replicated to several read-only DB replicas. From my understanding, the process of asynchronous ...
user avatar
-1 votes
2 answers
84 views

Difficulty understanding how denormalization results in more storage necessary

I am having difficulty understanding how denormalization results in more storage necessary. For example, let's say that in a normalized relation, there's Table 1 and Table 2. In order to join Table 1 ...
user avatar
  • 153
0 votes
1 answer
68 views

What considerations need to be made for choosing DBs in the OLAP portion of a scaling software system?

I'm trying to learn about the different types of database architectures and how their differences influence which one or combination of should be used in software system. I understand that OLTPs ...
user avatar
0 votes
1 answer
82 views

Benefits of Microservices processes split per kind? of process?

At our job we are having a heated discussion with the DevOps team on how to architect and modularize our microservices. We come from a 2-coupled-monoliths approach, so we are trying to avoid common ...
user avatar
  • 11
-1 votes
1 answer
51 views

Navigating a many-to-many relationship and determining price from a bulk API Call

I think it will help to give a little context, not only about the DB architecture but the actual product. I am building a business where we will be selling pallets of liquidated merchandise. Think a ...
user avatar
0 votes
0 answers
63 views

Prevent users from creating "global" database connections for a forking general purpose server?

I've created a general purpose scalable server in PHP. It's scalable in that the main process merely listens for incoming connections and immediately distributes them to forked child processes which ...
user avatar
-3 votes
2 answers
118 views

How Can I Create An Immutable Database That Can Be Truncated? [closed]

I have a requirement from a bank where they need to store data in a tamper proof way but at the same time they need to purge it after 7 years. The current process is they receive a letter and file it. ...
user avatar
0 votes
1 answer
121 views

How are Object Oriented Databases represented?

I have read many articles online but each one of them fail to give even a basic example on how an Object Oriented Database is stored and represented. For example each of the articles say that they are ...
user avatar
-3 votes
1 answer
103 views

How much access should a developer have to a Staging database [closed]

We want our developers to be able to diagnose production issues in staging. For the sake of this, we are planning to make an exact replica of production into staging. We are thinking however, if we ...
user avatar
  • 9
2 votes
1 answer
94 views

Should integration tests of a repository pattern use low-level ORM calls

Problem summary: In an application with wrapper methods over SQLAlchemy add() and query() methods, can integration tests that use the add() method wrapper use the query() method wrapper to validate ...
user avatar
  • 131
-2 votes
1 answer
122 views

Database index - Basic understanding

Trying to clarify my knowledge on databases and indexes. I'd like to know how exactly it works. So I have a few questions: When indexing a table over column or set of columns, a new table is created ...
user avatar
  • 113
9 votes
5 answers
739 views

Today I wrote "tests" for the testing code. Was it the right thing? Is it a smell?

My application has big integration with the database. Classes that use the database are very crucial for the system, so I write small unit tests for classes that I call Repository. The reason behind ...
user avatar
  • 695
1 vote
1 answer
173 views

How to store a huge volume of time-series datapoints in an efficient way?

We have an application producing 5k-10k datapoints per second. Each datapoint has more than one metric, alongside its time of creation. We are looking for an efficient, scalable way to store this huge ...
user avatar
  • 127
1 vote
2 answers
963 views

To which Clean Architecture layer should repositories implementations belong?

It's very common to see this use of repository in projects using clean architecture: interface Hero { } interface HeroRepository { findById(id: number): Hero; } class FetchHeroUseCase { ...
user avatar
1 vote
1 answer
88 views

Database Design, Associative Entities

I'm learning databases and I'm designing an ERD based on simple banking processes. I'm not sure if my design of the entity relationship diagram is correct. Thought Process A customer can open 1 : N ...
user avatar
  • 27
0 votes
1 answer
111 views

How sharding works when you outgrow the current shard architecture?

Looking at sharding techniques, you basically have hash-based or range-based versions. Hash based is more random, range-based is more heuristic based. Say you initially have 2 shards (separate ...
user avatar
  • 2,412
2 votes
3 answers
252 views

Should you use interfaces or classes in database entities?

Let's say I'm making a library database with books and authors, and I want to model the domain with interfaces, should I also specify the relations in the interfaces? For example, if I have the ...
user avatar
-2 votes
1 answer
52 views

Should cached data be kept separate from primary tables?

I have a table in an SQL database for storing information about some business object that looks something like this (details changed so as not to give away what company I work for): CREATE TABLE ...
user avatar
0 votes
2 answers
161 views

Share data between users without a database, php [closed]

How would I create for example a live chat, where you "post" a piece of text and it is displayed to other users in a "chat room", without the need of permanently storing it. My ...
user avatar
  • 23
-3 votes
1 answer
45 views

Ways & Methods switch between MongoDB Database after selecting a country. (MERNG)

I have a concept in which a user selects a country, which will either filter or go to that database; I was thinking of alternatives or options that would help in the process. These are my first plans ...
user avatar
0 votes
2 answers
116 views

When updating a model on a RESTful API, should there be an update endpoint per field? or one endpoint for the model?

For a RESTful API, consider a model schema as follows: MyCoolObject { field_a field_b field_c } Is it better to create one update endpoint to update one or many fields on the model? Or create ...
user avatar
-1 votes
2 answers
92 views

Database responsibility segregation

I have the following architecture: Database (Postgres), Batch(java app - long running operations that delete, update, insert lots of data in the database), Rest API(java app - provides rest services ...
user avatar
2 votes
4 answers
158 views

How to design around webapp Out of Memory crashing on export from SQL DB to Excel?

I have a monolith CRUD application with a feature - possibility to export some DB entires to an Excel file. The export is done within a background task. Often, however, when user presses the button ...
user avatar
  • 75
0 votes
4 answers
327 views

HOWTO: Full-text search over an encrypted database?

I want to adhere to the best practices and obviously encrypt the data sent to me by user. And I also want to provide the full-text search feature to the user (user can only search their own data, if ...
user avatar
  • 75
-3 votes
1 answer
120 views

Most efficient way to get average data for thousands of records from the database [closed]

I have a database that keeps a temperature readouts, which are being added - let's say - every minute: +----+---------------------+-------------+-----------+ | id | timestamp | temperature | ...
user avatar
  • 149
0 votes
0 answers
21 views

DRF API - using Multitable Inheritance vs "profiles"

I'm looking forward to your opinion on how to do this. This is a backend that will be served as a Django Rest Framework API. There will be multiple types of properties. Properties have approximately ...
user avatar
  • 119
1 vote
1 answer
32 views

Alternatives to ActiveX Data Objects for Serverless Implementations?

I have a few different web front ends that I use ActiveX Data Objects and ADODB connections to query an Access Database on a network drive. Usually there's some JavaScript logic that helps build the ...
user avatar
0 votes
1 answer
41 views

Design approach where to persist autocompletion suggestions

I am currently planning a movie recommendation application. Within this context, the user can fill out a form including different information (e.g. title, description, etc.). The new entry gets then ...
user avatar
0 votes
1 answer
146 views

Low Latency for multi-million record database in a web app [closed]

As a forewarning, this is for a school project with other school teammates, so my question is just for the sake of learning best practices. We are creating a recommendation system web app which ...
user avatar
0 votes
1 answer
53 views

Data Class: spread boilerplate for each different database over multiple class copies, or add complexity to original class?

I'm creating a prototype mobile app that will store data locally in an sqLite database and also on the cloud (CloudKit for now). The cloud storage uses a certain way of storing the data, and sqlite ...
user avatar
  • 1,477
1 vote
1 answer
71 views

An effective database migration pattern for many instances of the same app over a long period of time

How do you handle migrations to your database after a long period of time? I'm using fluent migrator, and after some months or years, the number of migrations could be (and has been in previous ...
user avatar
1 vote
2 answers
59 views

Compare diff between similar objects stored in two different databases and delete the missing records from the later

We have a legacy system (app 1) that has millions of records of an object Project. It has ProjectId and OrganizationId columns along with other properties. A recently added service (app 2) gets the ...
user avatar
  • 19
0 votes
0 answers
47 views

How to properly ingest mutable data into Snowflake?

Overview My app produces 100-1000 files of about 1 MB, per minute. And each file represents one single record. The records are json elements with dynamic structure. They are savegames from a videogame....
user avatar
  • 101
-1 votes
2 answers
107 views

Should I make all my business logic depend on the data that comes from the cache?

One of my web services hits a few hundred thousand visits per day where most of the operations are CRUD and we are thinking to get a cache sitting in front of our SQL database. I have had implemented ...
user avatar
  • 107
-1 votes
1 answer
73 views

Design approach to decouple DB write to reduce application latency

I want to use DynamoDB to save application operation data at the end of a session for auditing purposes without affecting the actual application latency. Because this date is only for auditing so we ...
user avatar
-1 votes
1 answer
30 views

CRDTs and merge conflicts in DB design

I face this problem with software development quite often, but I don't really know the name of this issue. It sounds quite common, though. Two nodes (a computer and a mobile phone, for example) share ...
user avatar
  • 155
3 votes
3 answers
547 views

How much should I avoid making internal communication?

I'm trying to figure out the best practice for querying databases or other services. If I can make one large query to a database once and sort the data in the backend should I do that over making many ...
user avatar
1 vote
2 answers
172 views

Minimizing queries on a database

I am working with a large dataset (~5GB) stored in a Google BigQuery database. My goal is to create a web app which works a little bit like an Excel spreadsheet where users start off by seeing the ...
user avatar

1
2 3 4 5
41