Questions tagged [database]
This tag is for general database questions. If you question is specific to SQL or NoSQL, use the corresponding tags instead.
2,030
questions
0
votes
3
answers
153
views
Writing a growing simulation that scales
Or ways to get better run time than a nested loop?
How would you design this?
I hope this is the place to ask semi-vague software engineering questions.
My simple simulation is like this:
There is a ...
- 119
1
vote
1
answer
109
views
Database design for a reccomendation system
I am working on a project which will have a recommendation system and I am in the database design phase, I have a good foundation in programming but databases is a new topic for me, I have designed a ...
- 11
-3
votes
0
answers
27
views
Structuring a database for an investment portfolio?
would really appreciate some general advice on the structure of an app i'm building.
I'm essentially trying to re-create an investment portfolio app using a MERN stack. Something very similar to ...
- 95
-1
votes
2
answers
127
views
Should I store non-editable items lists in a database?
Relational database is mostly used for storing editable user data.
But how about storing non-editable lists of information? Possible examples are:
GUI-related: main menu items, selectbox options, ...
- 109
1
vote
2
answers
109
views
Making data available for multiple microservices
We have a design challenge here for a project we are working on and I wonder if folks from the community can provide some guidance:
Our product is built in a microservices structure. So we have ...
- 111
0
votes
2
answers
75
views
Is it a good idea to use database sequential id as a key in external key-value storage?
I use Postgres database to store user information. Some of that information is sensitive so it was decided to store that information in Vault. The user table in Postgres has id's generated by a ...
- 145
1
vote
0
answers
31
views
Backend Design: System requires both ElasticSearch and MySQL. In which scenario we fetch data from MySQL or ElasticSearch?
So, one of the backend system, having 5 resource types (tables) and some foreign keys as well, requires full text support for two of the tables.
Also, for some use-cases, we require inner-joins for ...
- 121
0
votes
3
answers
119
views
Can you suggest ways to assure client data confidentiality in a SaaS product managed by yourself?
I am designing a multitenant SaaS platform where the database will hold information for different clients.
The application consists of an Angular frontend and a REST API Backend.
Some of the clients ...
- 367
1
vote
1
answer
77
views
Contextual Implementation Overhead vs Redundancy
I have an application that has use cases relative to where it is used. A local client and server, where all the server data is in a single location and the client queries that server knowing it is ...
- 119
16
votes
5
answers
4k
views
Why would an "articles" table (with no relationship to any other table) have a primary key?
Let's say that I have an articles table that have two columns: title and content, and let's say that this articles table doesn't have any relationship with any other table.
Why would this articles ...
- 399
0
votes
1
answer
58
views
Synchronous replication of databases
Say we have a software system with a web server serving writes to a Master DB. This Master DB is then replicated to several read-only DB replicas.
From my understanding, the process of asynchronous ...
- 17
-1
votes
2
answers
84
views
Difficulty understanding how denormalization results in more storage necessary
I am having difficulty understanding how denormalization results in more storage necessary.
For example, let's say that in a normalized relation, there's Table 1 and Table 2. In order to join Table 1 ...
- 153
0
votes
1
answer
69
views
What considerations need to be made for choosing DBs in the OLAP portion of a scaling software system?
I'm trying to learn about the different types of database architectures and how their differences influence which one or combination of should be used in software system.
I understand that OLTPs ...
- 17
0
votes
1
answer
82
views
Benefits of Microservices processes split per kind? of process?
At our job we are having a heated discussion with the DevOps team on how to architect and modularize our microservices. We come from a 2-coupled-monoliths approach, so we are trying to avoid common ...
- 11
-1
votes
1
answer
51
views
Navigating a many-to-many relationship and determining price from a bulk API Call
I think it will help to give a little context, not only about the DB architecture but the actual product. I am building a business where we will be selling pallets of liquidated merchandise. Think a ...
- 109
0
votes
0
answers
63
views
Prevent users from creating "global" database connections for a forking general purpose server?
I've created a general purpose scalable server in PHP. It's scalable in that the main process merely listens for incoming connections and immediately distributes them to forked child processes which ...
- 576
-3
votes
2
answers
118
views
How Can I Create An Immutable Database That Can Be Truncated? [closed]
I have a requirement from a bank where they need to store data in a tamper proof way but at the same time they need to purge it after 7 years.
The current process is they receive a letter and file it.
...
- 111
0
votes
1
answer
121
views
How are Object Oriented Databases represented?
I have read many articles online but each one of them fail to give even a basic example on how an Object Oriented Database is stored and represented. For example each of the articles say that they are ...
- 17
-3
votes
1
answer
103
views
How much access should a developer have to a Staging database [closed]
We want our developers to be able to diagnose production issues in staging. For the sake of this, we are planning to make an exact replica of production into staging. We are thinking however, if we ...
- 9
2
votes
1
answer
99
views
Should integration tests of a repository pattern use low-level ORM calls
Problem summary:
In an application with wrapper methods over SQLAlchemy add() and query() methods, can integration tests that use the add() method wrapper use the query() method wrapper to validate ...
- 131
-2
votes
1
answer
122
views
Database index - Basic understanding
Trying to clarify my knowledge on databases and indexes. I'd like to know how exactly it works. So I have a few questions:
When indexing a table over column or set of columns, a new table is created ...
- 113
9
votes
5
answers
742
views
Today I wrote "tests" for the testing code. Was it the right thing? Is it a smell?
My application has big integration with the database. Classes that use the database are very crucial for the system, so I write small unit tests for classes that I call Repository. The reason behind ...
- 695
1
vote
1
answer
176
views
How to store a huge volume of time-series datapoints in an efficient way?
We have an application producing 5k-10k datapoints per second. Each datapoint has more than one metric, alongside its time of creation.
We are looking for an efficient, scalable way to store this huge ...
- 127
1
vote
2
answers
1k
views
To which Clean Architecture layer should repositories implementations belong?
It's very common to see this use of repository in projects using clean architecture:
interface Hero { }
interface HeroRepository {
findById(id: number): Hero;
}
class FetchHeroUseCase {
...
1
vote
1
answer
88
views
Database Design, Associative Entities
I'm learning databases and I'm designing an ERD based on simple banking processes. I'm not sure if my design of the entity relationship diagram is correct.
Thought Process
A customer can open 1 : N ...
- 27
0
votes
1
answer
111
views
How sharding works when you outgrow the current shard architecture?
Looking at sharding techniques, you basically have hash-based or range-based versions. Hash based is more random, range-based is more heuristic based. Say you initially have 2 shards (separate ...
- 2,412
2
votes
3
answers
255
views
Should you use interfaces or classes in database entities?
Let's say I'm making a library database with books and authors, and I want to model the domain with interfaces, should I also specify the relations in the interfaces?
For example, if I have the ...
- 213
-2
votes
1
answer
52
views
Should cached data be kept separate from primary tables?
I have a table in an SQL database for storing information about some business object that looks something like this (details changed so as not to give away what company I work for):
CREATE TABLE ...
- 318
0
votes
2
answers
161
views
Share data between users without a database, php [closed]
How would I create for example a live chat, where you "post" a piece of text and it is displayed to other users in a "chat room", without the need of permanently storing it.
My ...
- 23
-3
votes
1
answer
45
views
Ways & Methods switch between MongoDB Database after selecting a country. (MERNG)
I have a concept in which a user selects a country, which will either filter or go to that database; I was thinking of alternatives or options that would help in the process. These are my first plans ...
- 125
0
votes
2
answers
127
views
When updating a model on a RESTful API, should there be an update endpoint per field? or one endpoint for the model?
For a RESTful API, consider a model schema as follows:
MyCoolObject {
field_a
field_b
field_c
}
Is it better to create one update endpoint to update one or many fields on the model? Or create ...
- 109
-1
votes
2
answers
92
views
Database responsibility segregation
I have the following architecture: Database (Postgres), Batch(java app - long running operations that delete, update, insert lots of data in the database), Rest API(java app - provides rest services ...
- 107
2
votes
4
answers
158
views
How to design around webapp Out of Memory crashing on export from SQL DB to Excel?
I have a monolith CRUD application with a feature - possibility to export some DB entires to an Excel file. The export is done within a background task.
Often, however, when user presses the button ...
- 75
0
votes
4
answers
332
views
HOWTO: Full-text search over an encrypted database?
I want to adhere to the best practices and obviously encrypt the data sent to me by user. And I also want to provide the full-text search feature to the user (user can only search their own data, if ...
- 75
-3
votes
1
answer
120
views
Most efficient way to get average data for thousands of records from the database [closed]
I have a database that keeps a temperature readouts, which are being added - let's say - every minute:
+----+---------------------+-------------+-----------+
| id | timestamp | temperature | ...
- 149
0
votes
0
answers
21
views
DRF API - using Multitable Inheritance vs "profiles"
I'm looking forward to your opinion on how to do this. This is a backend that will be served as a Django Rest Framework API.
There will be multiple types of properties. Properties have approximately ...
- 119
1
vote
1
answer
32
views
Alternatives to ActiveX Data Objects for Serverless Implementations?
I have a few different web front ends that I use ActiveX Data Objects and ADODB connections to query an Access Database on a network drive. Usually there's some JavaScript logic that helps build the ...
0
votes
1
answer
41
views
Design approach where to persist autocompletion suggestions
I am currently planning a movie recommendation application. Within this context, the user can fill out a form including different information (e.g. title, description, etc.). The new entry gets then ...
- 101
0
votes
1
answer
146
views
Low Latency for multi-million record database in a web app [closed]
As a forewarning, this is for a school project with other school teammates, so my question is just for the sake of learning best practices. We are creating a recommendation system web app which ...
- 103
0
votes
1
answer
53
views
Data Class: spread boilerplate for each different database over multiple class copies, or add complexity to original class?
I'm creating a prototype mobile app that will store data locally in an sqLite database and also on the cloud (CloudKit for now). The cloud storage uses a certain way of storing the data, and sqlite ...
- 1,476
1
vote
1
answer
73
views
An effective database migration pattern for many instances of the same app over a long period of time
How do you handle migrations to your database after a long period of time?
I'm using fluent migrator, and after some months or years, the number of migrations could be (and has been in previous ...
- 144
1
vote
2
answers
59
views
Compare diff between similar objects stored in two different databases and delete the missing records from the later
We have a legacy system (app 1) that has millions of records of an object Project. It has ProjectId and OrganizationId columns along with other properties. A recently added service (app 2) gets the ...
- 19
0
votes
0
answers
47
views
How to properly ingest mutable data into Snowflake?
Overview
My app produces 100-1000 files of about 1 MB, per minute.
And each file represents one single record. The records are json elements with dynamic structure. They are savegames from a videogame....
- 101
-1
votes
2
answers
107
views
Should I make all my business logic depend on the data that comes from the cache?
One of my web services hits a few hundred thousand visits per day where most of the operations are CRUD and we are thinking to get a cache sitting in front of our SQL database.
I have had implemented ...
- 107
-1
votes
1
answer
73
views
Design approach to decouple DB write to reduce application latency
I want to use DynamoDB to save application operation data at the end of a session for auditing purposes without affecting the actual application latency. Because this date is only for auditing so we ...
-1
votes
1
answer
30
views
CRDTs and merge conflicts in DB design
I face this problem with software development quite often, but I don't really know the name of this issue. It sounds quite common, though.
Two nodes (a computer and a mobile phone, for example) share ...
- 155
3
votes
3
answers
548
views
How much should I avoid making internal communication?
I'm trying to figure out the best practice for querying databases or other services.
If I can make one large query to a database once and sort the data in the backend should I do that over making many ...
- 141
1
vote
2
answers
172
views
Minimizing queries on a database
I am working with a large dataset (~5GB) stored in a Google BigQuery database. My goal is to create a web app which works a little bit like an Excel spreadsheet where users start off by seeing the ...
- 19
2
votes
0
answers
167
views
System Design: Chat application in a distributed system context
I am currently working on a side project so I can learn new technologies and understand how the components fit together. For this project, I decided that I wanted to work on Chat Applications in the ...
- 21
0
votes
1
answer
95
views
Syncing and matching local records to external database that behaves like a black hole
Black hole problem
I am working on matching and daily syncing of our internal data records to an external database that behaves like a black hole, meaning, we cannot view nor extract any data from it ...
- 9