Questions tagged [database]
This tag is for general database questions. If you question is specific to SQL or NoSQL, use the corresponding tags instead.
2,036
questions
-1
votes
0
answers
43
views
Designing APIs that don't expose database primary keys
People say "don't expose primary keys from the database in your API" because its a major security leak, so I'm trying to come up with a way for:
RESTful HTTP requests to reference server-...
-1
votes
1
answer
21
views
Picking data store for Location Tracking Based Application
I have one design problem for one of my projects based on location tracking. The scenario is I have multiple devices(for some business) all over the city and have GPS installed inside, the device is ...
0
votes
1
answer
34
views
Mixing static referred and dynamic data in database
I’m making an inventory for a game using nosql (mongo). I made some ItemType records (contains common data like icon, name, weight) and some ItemInstance records (representing specific items, ...
- 111
0
votes
1
answer
52
views
Database design - how to model database tables for similar objects - booleans / enums / inheritance / archived vs active
I only recently started to work with backends. My database is supposed to store information about maven-like artifacts. There are going to be applications and configuration artifacts. Now those ...
- 107
0
votes
0
answers
41
views
Database versioning when testing layered design
Let's say there are two layers of an application:
Business logic layer
Data access layer
Assume integration tests need to be written also for the Business logic layer. Alas tests that run against ...
0
votes
1
answer
52
views
Form builder app database design adding versioning
i'm making a custom form builder app similar to google forms or wufoo and i've completed the basic database design. However there is one thing i can't figure out. I'd like to keep track of version of ...
- 109
0
votes
1
answer
134
views
How to create financial reports with detailed information
I'm working on an application where it has financial control for the incomes and expenses of the company. These informations can be controled by accounts (bank accounts - can be 1 or more), categories ...
- 275
0
votes
3
answers
164
views
Writing a growing simulation that scales
Or ways to get better run time than a nested loop?
How would you design this?
I hope this is the place to ask semi-vague software engineering questions.
My simple simulation is like this:
There is a ...
- 119
0
votes
1
answer
121
views
Database design for a reccomendation system
I am working on a project which will have a recommendation system and I am in the database design phase, I have a good foundation in programming but databases is a new topic for me, I have designed a ...
- 9
-1
votes
2
answers
141
views
Should I store non-editable items lists in a database?
Relational database is mostly used for storing editable user data.
But how about storing non-editable lists of information? Possible examples are:
GUI-related: main menu items, selectbox options, ...
- 109
1
vote
2
answers
135
views
Making data available for multiple microservices
We have a design challenge here for a project we are working on and I wonder if folks from the community can provide some guidance:
Our product is built in a microservices structure. So we have ...
- 111
0
votes
2
answers
77
views
Is it a good idea to use database sequential id as a key in external key-value storage?
I use Postgres database to store user information. Some of that information is sensitive so it was decided to store that information in Vault. The user table in Postgres has id's generated by a ...
- 145
1
vote
0
answers
33
views
Backend Design: System requires both ElasticSearch and MySQL. In which scenario we fetch data from MySQL or ElasticSearch?
So, one of the backend system, having 5 resource types (tables) and some foreign keys as well, requires full text support for two of the tables.
Also, for some use-cases, we require inner-joins for ...
- 121
0
votes
3
answers
121
views
Can you suggest ways to assure client data confidentiality in a SaaS product managed by yourself?
I am designing a multitenant SaaS platform where the database will hold information for different clients.
The application consists of an Angular frontend and a REST API Backend.
Some of the clients ...
- 367
1
vote
1
answer
83
views
Contextual Implementation Overhead vs Redundancy
I have an application that has use cases relative to where it is used. A local client and server, where all the server data is in a single location and the client queries that server knowing it is ...
- 121
16
votes
5
answers
4k
views
Why would an "articles" table (with no relationship to any other table) have a primary key?
Let's say that I have an articles table that have two columns: title and content, and let's say that this articles table doesn't have any relationship with any other table.
Why would this articles ...
- 399
0
votes
1
answer
59
views
Synchronous replication of databases
Say we have a software system with a web server serving writes to a Master DB. This Master DB is then replicated to several read-only DB replicas.
From my understanding, the process of asynchronous ...
- 17
-1
votes
2
answers
85
views
Difficulty understanding how denormalization results in more storage necessary
I am having difficulty understanding how denormalization results in more storage necessary.
For example, let's say that in a normalized relation, there's Table 1 and Table 2. In order to join Table 1 ...
- 153
0
votes
1
answer
71
views
What considerations need to be made for choosing DBs in the OLAP portion of a scaling software system?
I'm trying to learn about the different types of database architectures and how their differences influence which one or combination of should be used in software system.
I understand that OLTPs ...
- 17
0
votes
1
answer
83
views
Benefits of Microservices processes split per kind? of process?
At our job we are having a heated discussion with the DevOps team on how to architect and modularize our microservices. We come from a 2-coupled-monoliths approach, so we are trying to avoid common ...
- 11
-1
votes
1
answer
51
views
Navigating a many-to-many relationship and determining price from a bulk API Call
I think it will help to give a little context, not only about the DB architecture but the actual product. I am building a business where we will be selling pallets of liquidated merchandise. Think a ...
- 109
0
votes
0
answers
67
views
Prevent users from creating "global" database connections for a forking general purpose server?
I've created a general purpose scalable server in PHP. It's scalable in that the main process merely listens for incoming connections and immediately distributes them to forked child processes which ...
- 576
-3
votes
2
answers
124
views
How Can I Create An Immutable Database That Can Be Truncated? [closed]
I have a requirement from a bank where they need to store data in a tamper proof way but at the same time they need to purge it after 7 years.
The current process is they receive a letter and file it.
...
- 111
0
votes
1
answer
123
views
How are Object Oriented Databases represented?
I have read many articles online but each one of them fail to give even a basic example on how an Object Oriented Database is stored and represented. For example each of the articles say that they are ...
- 17
-3
votes
1
answer
110
views
How much access should a developer have to a Staging database [closed]
We want our developers to be able to diagnose production issues in staging. For the sake of this, we are planning to make an exact replica of production into staging. We are thinking however, if we ...
- 9
2
votes
1
answer
126
views
Should integration tests of a repository pattern use low-level ORM calls
Problem summary:
In an application with wrapper methods over SQLAlchemy add() and query() methods, can integration tests that use the add() method wrapper use the query() method wrapper to validate ...
- 131
-2
votes
1
answer
122
views
Database index - Basic understanding
Trying to clarify my knowledge on databases and indexes. I'd like to know how exactly it works. So I have a few questions:
When indexing a table over column or set of columns, a new table is created ...
- 113
9
votes
5
answers
755
views
Today I wrote "tests" for the testing code. Was it the right thing? Is it a smell?
My application has big integration with the database. Classes that use the database are very crucial for the system, so I write small unit tests for classes that I call Repository. The reason behind ...
- 695
1
vote
1
answer
180
views
How to store a huge volume of time-series datapoints in an efficient way?
We have an application producing 5k-10k datapoints per second. Each datapoint has more than one metric, alongside its time of creation.
We are looking for an efficient, scalable way to store this huge ...
- 127
1
vote
2
answers
2k
views
To which Clean Architecture layer should repositories implementations belong?
It's very common to see this use of repository in projects using clean architecture:
interface Hero { }
interface HeroRepository {
findById(id: number): Hero;
}
class FetchHeroUseCase {
...
1
vote
1
answer
89
views
Database Design, Associative Entities
I'm learning databases and I'm designing an ERD based on simple banking processes. I'm not sure if my design of the entity relationship diagram is correct.
Thought Process
A customer can open 1 : N ...
- 27
0
votes
1
answer
112
views
How sharding works when you outgrow the current shard architecture?
Looking at sharding techniques, you basically have hash-based or range-based versions. Hash based is more random, range-based is more heuristic based. Say you initially have 2 shards (separate ...
- 2,408
2
votes
3
answers
290
views
Should you use interfaces or classes in database entities?
Let's say I'm making a library database with books and authors, and I want to model the domain with interfaces, should I also specify the relations in the interfaces?
For example, if I have the ...
- 213
-2
votes
1
answer
56
views
Should cached data be kept separate from primary tables?
I have a table in an SQL database for storing information about some business object that looks something like this (details changed so as not to give away what company I work for):
CREATE TABLE ...
- 318
0
votes
2
answers
161
views
Share data between users without a database, php [closed]
How would I create for example a live chat, where you "post" a piece of text and it is displayed to other users in a "chat room", without the need of permanently storing it.
My ...
- 23
-3
votes
1
answer
47
views
Ways & Methods switch between MongoDB Database after selecting a country. (MERNG)
I have a concept in which a user selects a country, which will either filter or go to that database; I was thinking of alternatives or options that would help in the process. These are my first plans ...
- 125
0
votes
2
answers
210
views
When updating a model on a RESTful API, should there be an update endpoint per field? or one endpoint for the model?
For a RESTful API, consider a model schema as follows:
MyCoolObject {
field_a
field_b
field_c
}
Is it better to create one update endpoint to update one or many fields on the model? Or create ...
- 109
-1
votes
2
answers
93
views
Database responsibility segregation
I have the following architecture: Database (Postgres), Batch(java app - long running operations that delete, update, insert lots of data in the database), Rest API(java app - provides rest services ...
- 107
2
votes
4
answers
162
views
How to design around webapp Out of Memory crashing on export from SQL DB to Excel?
I have a monolith CRUD application with a feature - possibility to export some DB entires to an Excel file. The export is done within a background task.
Often, however, when user presses the button ...
- 75
0
votes
4
answers
368
views
HOWTO: Full-text search over an encrypted database?
I want to adhere to the best practices and obviously encrypt the data sent to me by user. And I also want to provide the full-text search feature to the user (user can only search their own data, if ...
- 75
-3
votes
1
answer
122
views
Most efficient way to get average data for thousands of records from the database [closed]
I have a database that keeps a temperature readouts, which are being added - let's say - every minute:
+----+---------------------+-------------+-----------+
| id | timestamp | temperature | ...
- 149
0
votes
0
answers
21
views
DRF API - using Multitable Inheritance vs "profiles"
I'm looking forward to your opinion on how to do this. This is a backend that will be served as a Django Rest Framework API.
There will be multiple types of properties. Properties have approximately ...
- 119
1
vote
1
answer
38
views
Alternatives to ActiveX Data Objects for Serverless Implementations?
I have a few different web front ends that I use ActiveX Data Objects and ADODB connections to query an Access Database on a network drive. Usually there's some JavaScript logic that helps build the ...
0
votes
1
answer
42
views
Design approach where to persist autocompletion suggestions
I am currently planning a movie recommendation application. Within this context, the user can fill out a form including different information (e.g. title, description, etc.). The new entry gets then ...
- 101
0
votes
1
answer
153
views
Low Latency for multi-million record database in a web app [closed]
As a forewarning, this is for a school project with other school teammates, so my question is just for the sake of learning best practices. We are creating a recommendation system web app which ...
- 103
0
votes
1
answer
54
views
Data Class: spread boilerplate for each different database over multiple class copies, or add complexity to original class?
I'm creating a prototype mobile app that will store data locally in an sqLite database and also on the cloud (CloudKit for now). The cloud storage uses a certain way of storing the data, and sqlite ...
- 1,476
1
vote
1
answer
76
views
An effective database migration pattern for many instances of the same app over a long period of time
How do you handle migrations to your database after a long period of time?
I'm using fluent migrator, and after some months or years, the number of migrations could be (and has been in previous ...
- 144
1
vote
2
answers
59
views
Compare diff between similar objects stored in two different databases and delete the missing records from the later
We have a legacy system (app 1) that has millions of records of an object Project. It has ProjectId and OrganizationId columns along with other properties. A recently added service (app 2) gets the ...
- 19
0
votes
0
answers
49
views
How to properly ingest mutable data into Snowflake?
Overview
My app produces 100-1000 files of about 1 MB, per minute.
And each file represents one single record. The records are json elements with dynamic structure. They are savegames from a videogame....
- 101
-1
votes
2
answers
113
views
Should I make all my business logic depend on the data that comes from the cache?
One of my web services hits a few hundred thousand visits per day where most of the operations are CRUD and we are thinking to get a cache sitting in front of our SQL database.
I have had implemented ...
- 107