Questions tagged [database]

This tag is for general database questions; if you question is specific to SQL use that tag instead.

Filter by
Sorted by
Tagged with
20
votes
7answers
6k views

Advantages and disadvantages of using bit masks in database

Not so long ago I talked to my colleague and he was definitely against using bit masks because it is hard to understand all the values that are stored in the database. In my opinion it is not always a ...
1
vote
1answer
65 views

Architecture/algorithm for unusual recommendation system

There are thousands (or tens of thousands) of possible movie titles. New user enters my website and selects hundreds of titles that he likes. The only single goal of my website is to output the list ...
-1
votes
1answer
44 views

What is the best design for this use case regarding bidirectional relationship? [on hold]

For this question let's consider a system in which the users can match with each other (let's say for simplicity exactly like the dating apps). User1 matches User2 (and the other way around) only in ...
100
votes
9answers
43k views

How much business logic should the database implement?

I've worked in some projects where most of the business logic was implemented on the database (mostly through stored procedures). On the other side, I've heard from some fellow programmers that this ...
0
votes
1answer
29 views

Update items based on a read operations on same items is a bad practice?

DynamoDB is the database solution, I'm using to store my items. It's a NoSQL/Document database and I got a real problem on real-time updates of items. First, the saved format is like { expiredTime:...
0
votes
2answers
142 views

Is it a software engineer's responsibility to structure data as opposed to a DBA?

As a backend engineer at a small company, we're running into a question of whether we need a DBA or another backend engineer (SWE2). The reason why this is a question stems from a difficulty in ...
5
votes
0answers
325 views

Concurrent fault-safe data structure

I am building an application that aims to process ~10M data items per second. Each item is exactly 42 bytes small (including a sorting key) which means the total data rate will not be big 420 MB/s. ...
-1
votes
0answers
16 views

Best Database Architecture to Storage Vocational Test [closed]

I am with a college project and I would like to know the best way to storage a vocational test to support, in the future, big data analyses. Which NoSQL database is better? or which architecture i ...
13
votes
5answers
7k views

When should we use MongoDB?

MongoDB is a NoSQL database which I've found quite easy to use. Recently I had to develop a simple application which needed to collect some data using HTTP requests and store some results after ...
0
votes
1answer
32 views

Strategies for Rotating MYSQL Encryption Keys

I have a web app that connects to a MYSQL database which has some encrypted data in it. The encrypted data is encrypted using the the mysql AES_ENCRYPT function and a key. Now, this key has been ...
-1
votes
1answer
101 views

I have a data set of over a million addresses and I want to display the closest N locations to a given address or current location

I am a student working on a personal project which is essentially a location finder that will be on a website. I have a data set of over a million addresses and I want to display the closest N ...
0
votes
1answer
50 views

Data backend for a 'Nearby' places application

Context I am building an application for searching places 'Nearby' akin to Yelp/Google. Objective for this question Recommend a strategy to do paginated searches based on location and ask for ...
-1
votes
2answers
113 views

When self-referencing in a table, do you prefer `parent_id = null` or `parent_id = id` for a row referencing to itself?

For the discussion, I will provide my example. But I am interested in the broad guidelines. I save in my postgres DB legislation (law texts) composed of a single header (H), paragraphs (P), sub-...
0
votes
2answers
4k views

Size of database a factor for Hibernate vs JDBC?

Do you know if the size of the database (number of tables used) is a factor when choosing between Hibernate and JDBC? Why or why not? In my particular case, I am evaluating Hibernate and JDBC for ...
0
votes
1answer
113 views

Database Design Structure

I am building a WebApp which has a list of Assets. I have a peculiar requirement and I am unsure of the best way to store the data in SQL. Requirement: My Asset Headers currently have a relation to ...
0
votes
0answers
57 views

Logs/audits in DB - handling huge row number [closed]

We currently have a product which has audit table which is used for storing most of the user CRUD actions on the system. For example "user X created Y in module Z". Now the issue we're facing is that ...
0
votes
0answers
18 views

Updating an entity and many relationships: database vs application layer

We are working on a transformation project It has records in table entityA and entityB, which has a relation entityAB. We need to delete records in table entityA and recreate them with some other ...
0
votes
1answer
37 views

Bridge/Join Table for a Many to Many Relationship between One Table and Multiple Others

Background I have a relational database an I want to have a table that contains flags. I want multiple tables in my database to have a many to many relationship with the flag table. Options I ...
0
votes
1answer
172 views

Is it good practice to create a static class for database tables calculations?

I need to implement some system (using python pandas dataframe in my case) that parses raw data, then adds on calculated data, and then validates that calculated data (boolean output on some columns) ...
0
votes
1answer
51 views

How to store data which can be a foreign key or free text

I am not sure how to explain my problem, but basically I have an input field which lets you enter member's name or select one from a dropdown list. If the user has selected the person from the list ...
0
votes
1answer
72 views

Deciding which microservice should write to the database

Let's assume the following architecture where EventCreatorApp creates an Event which is logged to DB and sent to WorkerApp for processing WorkerApp processes the event, handles failures, etc. ...
1
vote
1answer
39 views

how simulate sequential consumers in Kafka

I have kafka consumer that get data from kafka and do some process if some circumstances meet and send it for another consumer to do it's job. How can i build something like this? One way I think ...
0
votes
1answer
103 views

What to do with the unused identity after deleting a row?

I have a user table that has 4 rows with id = 1, 2, 3, 4. I delete the row with id = 2 and insert another user and the user will get id = 5. What will happen to the id = 2? Will it be used again or do ...
2
votes
1answer
71 views

Can I reduce locking for this scenario using Hibernate and H2

My Java application uses Hibernate and H2. It process music files within folders. Typically for a folder it would read the files in the folder and store each file as a Song class in database, then ...
5
votes
5answers
3k views

When keeping product price change history, what are the pros&cons of keeping track of price change events only?

When tracking something like pricing history, in my relational database, what is a recommended way to store pricing changes? For example I can: keep price change events only (aka a single date ...
0
votes
1answer
40 views

Temporary users to restrict db access of query?

I am new to database managment, and I am wondering if it is a good idea to create temporary users to restrict access to only a single table. Like having an user named _logs can only (read?) and ...
33
votes
3answers
31k views

Is there any reason to use varchar over text columns in a database?

Is varchar just a remnant from before text came around, or are there use cases where you would want to use a varchar? (Or char for that matter..) (I use Postgres and MySQL (MyISAM) daily, so those ...
129
votes
9answers
97k views

When would someone use MongoDB (or similar) over a Relational DBMS?

I'm a bit confused about the whole NoSQL thing and such. When would you choose to use something like MongoDB over something like Oracle or MySQL? I don't really understand the "difference" as far as ...
3
votes
3answers
414 views

Aggregation of data from two Microservices

I have two Microservices A and B. B Microservice has a large set of an entity called User. A Microservice stores the User entity in its own DB if User is configured by an agent. There is no flag ...
4
votes
2answers
111 views

Database structure for word co-occurrence frequencies in a large corpus

I would like to store the frequencies with which words co-occur with each other over a variety of contexts in a large (> 1 billion tokens) text corpus. I need to store the word pair, the type of co-...
1
vote
1answer
94 views

Multiple applications with single database for IoT

I'm using a raspberry pi 2B with raspbian lite. I have made a program that communicates with a LoRa module (RFM95, sx1276 chip) and collects data from a sensor. The data it collects, are stored in a ...
2
votes
1answer
31 views

Setup pipeline to analyze data stored in web app DB

Background: So there is a (Ruby) web app with a production Postgres DB (hosted in the cloud) I would like to run some machine learning algorithms in a Python setting on the production data and (...
6
votes
4answers
17k views

ERD: “many” vs “zero or many”/“one or many” crowfoot notation?

Background I saw this figure describing the different crowfoot notations used in ERD: I'm not able to find the difference between the "many" notation and the "zero or many. However I was able to find ...
4
votes
2answers
214 views

How to test database dependent functionalities?

I have written this function which checks if a user already exists in the application: async function ValidateUserExists(username, email){ if(!username || !email) throw new Error('Invalid number ...
2
votes
4answers
186 views

Database agnostic DAO (NoSQL + SQL)

Background While writing a new component, I m in middle of making a decision of SQL/NOSQL database (Mongo vs Mysql) for my storage layer. As of today, mysql seems to be a perfect fit for my use-case (...
0
votes
1answer
82 views

ER Diagram in Barker Notation

I engineered an ERD diagram using Barker's notation. The problem was to make a database for system of water pipes where one node is connected to another using a section (pipe), each node can have ...
2
votes
1answer
69 views

Allow client side direct access to the database [duplicate]

I am asking advice from the SQL graybeards. Because my following proposition goes against everything I have practiced. In this instance Let's assume we are developing a phone app. A social network ...
1
vote
0answers
116 views

How to store relational data belonging to different client customers?

My users will be businesses with a small number of accounts each: e.g. Business #1 with 3 users, Business #2 with 5 users, etc. I am trying to determine the best way to organise the relation (on ...
0
votes
0answers
27 views

ACL entity and microservices database relations

I would like to build a microservices architecture to handle the following scenario: I have a building. The building has some kinds of devices. And the devices can be used by the users. In the system, ...
2
votes
2answers
290 views

Should arbitrary numbers be stored as strings in a database?

I'm using Google as a login system for my latest project. So, my users table has a column named googleid, which stores Google's unique user id, which is a number, with a few dozen digits. For example, ...
1
vote
8answers
656 views

Is there a way to prevent developers leaking data while they are developing?

I am working with a startup company for a while now. They have been affected by some reckless developers in the past. One area they want a real and immediate improvement is when they assign a task to ...
8
votes
1answer
2k views

Making API calls with celery

I'm designing a system for a client where the requirements are: they upload a JSON file (one object/line) make a call to an API with the JSON object as the payload record the state (success/failure) ...
2
votes
1answer
149 views

Unit conversion of data

My team is developing a web application, where we have in the requirements to display the data in multiple unit which is set by the end user. Where should we write down the unit conversion logic, on ...
-3
votes
5answers
151 views

One Single Table or Two Smaller Tables

We currently have a Users table in the database, this has quite a few columns in it. Around 50% of the columns are barely used in the system (only on one or two pages). We've been discussing amongst ...
1
vote
1answer
81 views

Approach to handle network breakdowns in application dealing with database?

I have an application that writes data to database. In case of network breakdowns, the connection to database gets lost & my db queries freezes the whole application. So, in order to make it ...
1
vote
1answer
77 views

About databases in docker containers

I'm trying to understand containerized apps and databases and I'm trying to understand the microservice architecture using kubernetes. One thing that I couldn't get my mind convinced is the database ...
126
votes
17answers
12k views

How do you store “fuzzy dates” into a database?

This is a problem I've run into a few times. Imagine you have a record that you want to store into a database table. This table has a DateTime column called "date_created". This one particular record ...
18
votes
3answers
6k views

How to store ordered information in a Relational Database

I am trying to understand how to properly store ordered information in a relational database. An example: Say I have a Playlist, consisting of Songs. Inside my Relational Database, I have a table of ...
0
votes
1answer
41 views

How do database storage engines handle new columns?

I really don't know which StackExchange site this belongs on. It didn't seem to match the "on-topic" list of DBA.SE or SO.SE ... so please let me know if I should move it. I also couldn't find a ...
0
votes
1answer
147 views

When is the point where a JSON text file storage is not enough?

I guess it was Robert C. Martin who campaigned against using SQL in one of his apps, where he was just fine without DB and SQL initially. Later, thanks to the abstraction a DB implementation could be ...