Skip to main content

Questions tagged [database]

This tag is for general database questions. If you question is specific to SQL or NoSQL, use the corresponding tags instead.

Filter by
Sorted by
Tagged with
0 votes
0 answers
54 views

Storing SqlAlchemy models for database with >20 tables

I'm working on a pet project for a web shop that uses SQLAlchemy (SA) and connects to a database with over 25 tables (thus, I have 25+ SA models - one for each table). What would be the best practices ...
Max Azatian's user avatar
3 votes
3 answers
204 views

What is the difference between a key attribute store and a key value store?

Im running through a course on Cloud Academy - now known as QA - and it makes the statement that Azure Table Storage is a "Key/attribute store" and Redis cached is a "Key value store&...
user1605665's user avatar
2 votes
1 answer
147 views

How to design sharing feature properly in DB?

My DB has three tables: CREATE TABLE Users ( id INTEGER PRIMARY KEY, name TEXT NOT NULL ); CREATE TABLE Categories ( id INTEGER PRIMARY KEY, name TEXT NOT NULL, user_id INTEGER ...
Sergey K's user avatar
0 votes
1 answer
97 views

How to handle data when source of truth is through API

I am making a webapp that deals with money movement. All the financial actions are done through an API. For example, right now I can create an account for a user, add funds to their account, transfer ...
Mitchell Jeppson's user avatar
-1 votes
2 answers
134 views

Text files vs. MySQL Performance for Large Dataset

I am currently making a system for users to generate flashcards for Languages. This involves adding information such as the definition, pronunciation, and example sentences for a word. So far, I have ...
Jarvis Coghlin's user avatar
1 vote
1 answer
82 views

Single-source data warehouse permissions management

We (Data Platform team) are reviewing how we configure and apply permissions against our data warehouse objects, and I'm curious what tools or custom systems you might be using for this. For context ...
Zé Pinhao's user avatar
-2 votes
2 answers
164 views

When should I think about two databases (the first for reads, the second for writes) in CQRS?

I am not asking what are advantages of two databases - the first for reads and the second for writes. I am asking WHEN I should have two databases for reads and writes? Probably there is no simple ...
Aleksander Chelpski's user avatar
2 votes
1 answer
149 views

What is an optimal system design for tracking product views per user that is scalable?

I have a web application that contains products and users. There are 10,000+ products and 100,000+ users to give a sense of the scale that's required. For some application specific reasons, I need to ...
kitkat's user avatar
  • 29
0 votes
0 answers
52 views

Hash Indexes vs LSM trees with SSTables

I was reading Designing-Data Intensive Applications, and I am confused about the usage of LSM trees with SSTables. The author talks about Hash Indexes and log files (written as segments which are ...
Ufder's user avatar
  • 244
7 votes
5 answers
376 views

What can I do to get a message processor to slow down the rate of writes that it is making to a database?

We have this architecture: queue -> message processor (horizontal scaling) -> RDBMS Sometimes external systems dump 10k messages onto the queue and the message processor of course dutifully ...
jcollum's user avatar
  • 229
-1 votes
2 answers
117 views

How to handle complex logic, avoiding recalculation for performance reasons?

let's say we're building an Ecommerce Marketplace. We have Sellers and each Seller has Products. We want to display a list of available Products across all Sellers to Buyers. But, we only need to ...
Staskij's user avatar
  • 21
0 votes
1 answer
82 views

Referencing objects created during seeding by ID

I'm building a software that allows users to extract analytics from a data source. A subset of data periodically is extracted from the data source and stored in a separate database for ease of access (...
Marceli Wac's user avatar
2 votes
1 answer
182 views

Designing a flexible and comprehensive restaurant menu database schema?

I am working on developing a MySQL database schema for a restaurant menu system inspired by platforms like DoorDash, UberEats, and Yelp. My aim is to design a schema that is flexible and can handle ...
tony's user avatar
  • 145
1 vote
2 answers
100 views

Dynamic web application hitting database on front page load to fetch profile image

Folks, I am in middle of writing a web application (Python/Flask) where home page has user profile image in the navbar which is coming from a database (blob), I am wondering if this is a good practice....
afsar's user avatar
  • 19
10 votes
8 answers
6k views

What is the advantage of log file rotation based on file size?

I understand that log file rotation is changing the log file you used when (1) one gets big enough or (2) at EOD, but I'm not sure I understand the reason for (1). I have never had any issues with ...
user129393192's user avatar
1 vote
1 answer
114 views

Scaling Heavy Writes When Fetching Data From External API

I have a service that fetches financial data from an external APIs and parses and saves that data in a postgres DB. The data can be quite large in size, our containers have crashed quite a lot due to ...
Sachin Saini's user avatar
2 votes
7 answers
422 views

Why does "a consistent, understandable interface" differentiates database from global states?

According to some answers of How are globals any different from a database? that explains how database is different from global state: https://softwareengineering.stackexchange.com/a/319389 https://...
wcminipgasker2023's user avatar
11 votes
5 answers
3k views

Why split up data retrieved from a database into multiple endpoints, if we need ALL the data anyway?

I have a "Games" API which retrieves video game data from a large database. The /games endpoint returns some very basic information about the game, such as the title, description, etc. More ...
Yeager's user avatar
  • 121
1 vote
3 answers
402 views

What is the correct way to find the differences between 2 relational tables?

Typically the solution for comparing if 2 relational db tables (I am using AWS Athena) are equal is to do full outer join on all the columns but adding an extra column to each dataset that acts as a ...
Jeremy Fisher's user avatar
-3 votes
2 answers
109 views

Company and person or just contact table [closed]

Given the two models company and person I'm noticing duplicate fields i.e. zipcode, emailaddress for each. Yet, a short research reveals a reasonable popularity for separating those two entities into ...
jjk's user avatar
  • 545
2 votes
1 answer
127 views

Handling Back-dated Transactions and "Online" Transactions on ERP/Banking Systems

I was currently developing a PoC of a "core banking system" in my current company, a pawn shop that generally offers lending services to retail customer and deposit services. We've built a ...
jeremy__'s user avatar
0 votes
0 answers
40 views

Looking for patterns for instantaneous data consistency for DB transactions that span multiple schemas in a Aurora postgres instance

We are currently developing a product in the AWS environment that requires instantaneous data consistency for DB transactions that span multiple schemas in a Aurora postgres instance. Each schema will ...
Jeff Prisco's user avatar
2 votes
2 answers
195 views

When should I be worried of Time of check time of use vulnerabilities during database queries?

I am having difficulties on understanding when I should be worried about TOCTOU vulnerabilities and how to avoid them because yes, we can use database transactions but there are different level of ...
Alessandro's user avatar
1 vote
2 answers
175 views

Synchronize data between old on premise application and new cloud system

Scenario/Context There is an old monolithic desktop application (running locally) which uses ODBC to connect to SQL database (on premise). This desktop application should be replaced with a new web ...
Malte G's user avatar
  • 119
0 votes
1 answer
369 views

Clarification on using timestamp without time zone and LocalDateTime in UTC-centric systems

I'm seeking clarification regarding the usage of timestamp without time zone in a database and its corresponding Java type, LocalDateTime. The current setup I'm working on involves servers and (...
user avatar
-2 votes
1 answer
85 views

What is a route I could take when considering setting up a data warehouse for a small business? [closed]

I am working for a medium-sized development and construction company and we are looking to upgrade our data and BI. Currently, all project data is managed via Excel (data, project management, and ...
Robert Eldridge's user avatar
0 votes
0 answers
31 views

Sharing data through multiple environments (dev/test/stage/prod) for one usecase

I am looking for a concept that handles the needs of some data to be distributed in several environments (lets say we have dev, test, stage and prod). Example: We are setting up planned outages of ...
libik's user avatar
  • 320
20 votes
6 answers
7k views

How would you identify slow queries before reaching production?

Let's say you are running a query on your database. In development and staging, where the amount of data is small, everything looks good. Unit and integration tests are also fine and the logic is ...
riorio's user avatar
  • 489
1 vote
1 answer
159 views

Reserving Unique, Sequential Group Numbers in ASP.NET Core with EF Core and GraphQL

I am developing an API backend using ASP.NET Core Minimal API (.NET 7) with Entity Framework Core, HotChocolate (for GraphQL), and a Microsoft SQL Server database. I've encountered a specific ...
bealtis's user avatar
  • 19
1 vote
1 answer
116 views

Persisting data in a mobile app backed by a rest API back-end, advices [closed]

So I am working on this mobile application using MAUI, which is backed by a Rest API in ASP.NET Core. I want to be able to use the app offline, and the user can create/edit some data. This data is ...
Toto's user avatar
  • 155
0 votes
2 answers
386 views

Is it reliable to process millions of records with INSERT INTO SELECT directly in the database?

I have several million booking rows in a table and would now like to save the totals of the booking amounts in another table depending on the customer, account no., product, etc. Should this ...
root66's user avatar
  • 121
-1 votes
1 answer
190 views

Modeling a CSV file: What is the standard? Python or SQL?

I have a wide CSV file of about 350mb, and want to load it into a SQL database and properly model the data to make it easier to use for analysis. I could split the data into tables with python and ...
HappilyCoding's user avatar
0 votes
2 answers
813 views

Best approach to sync data between mobile app cache and the server db? [duplicate]

I have an offline-first mobile app. The user enters data, data is saved into local SQLite database. The server has Postgres database. I need to build a two-directional sync strategy that keeps data on ...
Andrey Gordeev's user avatar
1 vote
2 answers
289 views

High-cardinality UUID: how/where to store? [closed]

I have lots of devices, each identified by a unique UUID. For each device, I have one or more timeseries data (potentially up to millions of entries). What's the best way to store these entries? ...
persson's user avatar
  • 197
-1 votes
2 answers
248 views

Should I store large amounts of data in the properties of a node in a graph database?

I'm building a reddit clone on top of software that must incorporate a legacy neo4j graph database to track user actions. I also want to use it to store the comment/post data. Assuming this new app is ...
Austin Capobianco's user avatar
3 votes
2 answers
240 views

Microservices: separate db instance vs. separate db server

In a scenario with multiple teams, each overseeing a few microservices that collectively create a larger system, and where these services frequently use MongoDB databases, is it advisable for each ...
weno's user avatar
  • 281
-2 votes
2 answers
198 views

Designing a database schema for a website related to my town's history

I am working on digitizing old photographs, articles, ephemera, etc. related to my town's history, and I've been tasked to put these scans I've made onto a website for the public to see. There are ...
EdmundF's user avatar
  • 13
12 votes
5 answers
7k views

Is it an anti-pattern to create ORM entities based on existing database schema?

I maintain a large legacy app working with SQL database. The app uses raw SQL queries to query the database. Together with app rewrite I plan to introduce ORM to work with the database. The simplest ...
user14967413's user avatar
2 votes
3 answers
479 views

Dynamically transform EAV data into standard SQL table

I use MariaDB to store data using EAV model. I need EAV, because each attribute of each entity can have its own validity (validFrom, validTo). For example, I have entity person and I track changes ...
user14967413's user avatar
0 votes
1 answer
77 views

Design a sequential processing of records

About 10 to 15 records are processed per day with the time interval of 5 minutes between each record. System A inserts a record in DB and sends id of that record to active mq. System B Listener ...
sunleo's user avatar
  • 119
2 votes
2 answers
172 views

Pattern to transform and cache data from a DB?

We have a server that receives queries. We use the command pattern for this, e.g. q := NewQuery(database) q.Execute(request1) q.Execute(request2) The queries read a "model" from the ...
Maria Ines Parnisari's user avatar
12 votes
4 answers
6k views

Using three different database engines in the same application

Currently, I'm working on a graduation project about designing a Document Management System (DMS) that incorporates some functionalities as follows: It won't use folders to organize documents; ...
siunhanbumpbump's user avatar
0 votes
2 answers
169 views

API - What are some good strategies to limit hitting database on query-based searches?

I'm building an API that allows people to search for books (just for simplicity). There is an endpoint that takes a query string and returns the top 10 matched results. For example: /api/v1/book/find?...
Richard H. Nguyen's user avatar
15 votes
4 answers
5k views

Better solutions than joining table for Many to Many?

Lets say I have students and classes which are 2 entities. A student can take many classes and a class can have many students. This would be a many to many relationship. To solve this with an RDBMS my ...
Jeremy Fisher's user avatar
1 vote
2 answers
173 views

Handle data layer when using DDD

Since March I switched jobs and became the Tech Lead of two teams that work on a shared codebase and a separate codebase for "region" specific implementations. This team works using NestJS, ...
Matias's user avatar
  • 31
1 vote
2 answers
454 views

Restricting access to sensitive data in monorepo

Imagine that you have a large monorepo code base running as a monolith application. This application is backed by a database. Some of the data in the database is sensitive, so you want to restrict ...
Matt D's user avatar
  • 19
0 votes
3 answers
221 views

In data engineering, why is data integrity checked on the DW rather than on the data sources?

I'm a software developer and new to data engineering, so this may be a newbie question, but I'm wondering why data integrity checks (for instance, dbt tests) are ran on the data warehouse, rather than ...
samdouble's user avatar
  • 243
7 votes
3 answers
1k views

Prevent analytics from crashing the production database

At the core of my business (say, an online store) is a Postgres database that stores products and transactions. During day-to-day operations, the load on the database is not heavy. However, there is ...
firstname gklsodascb's user avatar
3 votes
1 answer
142 views

Should I use transactions in this scenario?

I am writing a API endpoint in NodeJs, The code is roughly like this: function myApi(myUserId, userIdToDelete){ if ( checkIfIAmAdmin(myUserId) ) { deleteUser(userIdToDelete); } } Now,...
Alessandro's user avatar
0 votes
1 answer
250 views

How can I trace back the original table of a column?

New to a software engeneer job and I am kinda blocked on the way to go with my new project To explain this in a scheme. I have a source of data in a custom app that is for most part of the time the ...
Dolotboy's user avatar
  • 111

1
2 3 4 5
43