Questions tagged [database]

This tag is for general database questions; if you question is specific to SQL use that tag instead.

132 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
7
votes
0answers
361 views

Concurrent fault-safe data structure

I am building an application that aims to process ~10M data items per second. Each item is exactly 42 bytes small (including a sorting key) which means the total data rate will not be big 420 MB/s. ...
7
votes
0answers
97 views

When connecting to a server using the DRDA protocol, is it true that the first Client-To-Server command MUST be EXCSAT chained with ACCSEC?

When connecting to a server using the DRDA protocol, is it true that the first Client-To-Server command MUST be EXCSAT chained with ACCSEC? I found 2 different answers when I googled it. If you look ...
5
votes
0answers
489 views

Class table inheritance using Slick: always an anti-pattern?

My application's model includes a number of entities that share many common properties, and are operated on by many of the same classes. If I was using an ORM, I'd define an abstract class with the ...
4
votes
1answer
414 views

Continuous Deployment Database Project VSTS

Looking for continuous deployment strategies regarding SQL server database projects targeting Azure SQL using VSTS. My scenario... Using VSTS for CI/CD Using a SQL Server Database Project to define ...
4
votes
0answers
221 views

Architecture: mobile apps for different clients with their own dbs different servers?

My general feeling is that this is a question which must have been answered but I am struggling to find resources for it. I know little about mobile app dev but my knowledge of web services is pretty ...
4
votes
0answers
571 views

Is Event sourcing a good fit for database sync (master-multiple-slaves)?

I'm looking for a good way to maintain in sync local sqlite databases that could be offline to a master one, for a invoice app. Then I read about EventSourcing and Log structured storage and wonder ...
4
votes
0answers
711 views

Database version management on mobile apps

I am about to start building a mobile (iOS/Android) app that uses a local database (probably SQLite). Since this is my first "real" mobile app, I am wondering what the standard approaches are to ...
4
votes
1answer
818 views

Design for syncing data from remote API

I am trying to find a robust and correct solution for syncing data from different sources to my local relational database. I have different sources of data for eg: CSV, REST API and probably network ...
3
votes
0answers
39 views

Designing provisioning strategies

I'm trying to redesign a provisioning application that administrates other application through their REST API. The application is quite simple, we have groups, we have users and roles and we have a ...
3
votes
0answers
194 views

Chronicle Queue: how to make a stack and how to limit disk usage

My server and several clients are different JVMs running on the same machine, they need to share persisted time series data: server reads a time series from somewhere, writes to a persisted embedded ...
3
votes
0answers
466 views

File upload sync for offline desktop application. (Electron)

We are making an offline desktop application in Electron that needs to sync when the user goes online. Sqlite is used as the database when offline and it gets synced to postgres on the server when ...
3
votes
1answer
203 views

Designing database job monitoring

I have several Oracle database where my in-house applications are running. Those applications use both dba_jobs and dba_scheduler_jobs. I want to write monitoring function: check_my_jobs which will ...
3
votes
0answers
788 views

Reading and saving big data to db

I have a method for reading data from file. The problem is how to handle files that are too big for a simple read and save to database? I was thinking about reading a chunk of it and saving it to ...
3
votes
0answers
612 views

How do I implement Socket.io Leaderboard using AWS Dynamo DB cost-effectively?

Every time a user submits a score, Socket.IO causes all the users get their leaderboard updated. This causes every single user connected through Socket.IO to read from the Dynamo DB database (every ...
3
votes
0answers
164 views

Are there downsides to writing PDO/mysqli statements into my code directly vs using my own redirect wrapper methods?

TL;DR: To access DB functionality in my codebase I currently use a custom layer throughout the code. The layer needs an upgrade in order to begin accepting prepared statements, which it currently ...
3
votes
0answers
223 views

Saving tabular, spreadsheet-like data in a relational database?

I'm hoping this question isn't too broad or too prone to opinionated answers, since I could really use some pointers. I'm trying to come up with a way to persist tabular, spreadsheet-like1, data in a ...
3
votes
0answers
415 views

Maximising data processing speed: in-memory or write to db first?

Currently working on a system which allows users to upload data files, which we then "clean" (i.e. process in various ways). Speed of processing and return to the user is an important consideration, ...
3
votes
0answers
89 views

Suggestions for Storing large collection of related words

I am trying to create a simple sentence generator that uses templates and a database of words. It will be a website where essentially a user could click a button to generate a sentence. For example: ...
3
votes
0answers
686 views

how to import large amounts of data without locking sqlite database

Background Information I have a web app that provides the user with the usual CRUD functionality for "widgets" let's say. But I also have give them the option of creating widgets in bulk via ...
3
votes
0answers
477 views

Syncing structured data and files in Android App with an ORM?

I am developing an Android App which I have developed for iOS before. The app requires that structured data and files are synced across android devices. In iOS, I was using Core Data and synced to ...
3
votes
1answer
633 views

Three phase commit protocol

I'm trying to understand the three phase commit protocol by reading the wiki page (http://en.wikipedia.org/wiki/Three-phase_commit_protocol#Motivation). In the description for the diagram I posted ...
3
votes
0answers
178 views

What are some important considerations when developing a DBaaS (DataBase as a Service)

I'm interested in creating a Database as a Service. I want users of the service to be able to do a few seemingly not very simple things. Sign up and be given access to a JSON datastore that they can ...
3
votes
2answers
318 views

Should audit records have their own id?

I'm creating audit tables to track changes made to some of my crucial tables and I'm wondering whether there's a point to history rows having their own id. Here's how my interfaces look like: public ...
2
votes
0answers
102 views

Is it possible to implement insert file operation in modern extent-based filesystems?

The problem was already discussed here. But there was not consensus on this topic. I have some thoughts on how insert operation can be implemented for some popular file systems. If FS has extent-...
2
votes
0answers
134 views

Data Types Based Database

I was wandering if there exists some kind of database that organises data based on some declared type rather than tables or collections. I'll try to explain better with an example. Let's say I have ...
2
votes
1answer
45 views

Data modeling user-object permissions, between claims and metadata

I've been thinking about how to map user claims to permissions in a way that scales from a read and write perspective, probably using a stand-alone service. What we have today doesn't scale for ...
2
votes
0answers
83 views

Data pipeline architecture: airflow triggered by message broker

Let us say we have: a web app with a Postgres DB that produces data over time, another DB optimized for analytics that we would like to populate over time. My goal is to build and monitor an ETL ...
2
votes
0answers
87 views

Idea on replacing DAO/Domain obj pattern

I know someone has probably already thought of this, but here is a way to replace DAOs and Domain objects: For example public class Bike { private int id; public Bike(String model) { //...
2
votes
0answers
326 views

Logging Database Changes

What is the proper way of storing a changelog, similar to git, for data in a database. Currently, I have it set up so that every line in a table is given a unique string id that is used to determine ...
1
vote
2answers
62 views

Separate database in microservice architecture

I'm trying to create project siteWeb and mobile app with clean Architecture and Microservices. create identity service with separate database A and make another api service with other database B. for ...
1
vote
0answers
53 views

Database vs Logs - Persistence approach evaluation

I have a node js app, in which I need to perform background jobs. My plan is to use a data structure which will contain all the jobs needs done. My design also suggests that this data structure needs ...
1
vote
2answers
105 views

which layer must check if something(like account etc) exist in database

I am implementing a web api and i have crated three layers,data access layer,business layer and presentation layer. In data access layer I have a method which create a user.I use a transaction which ...
1
vote
0answers
89 views

ACL entity and microservices database relations

I would like to build a microservices architecture to handle the following scenario: I have a building. The building has some kinds of devices. And the devices can be used by the users. In the system, ...
1
vote
0answers
121 views

How to store relational data belonging to different client customers?

My users will be businesses with a small number of accounts each: e.g. Business #1 with 3 users, Business #2 with 5 users, etc. I am trying to determine the best way to organise the relation (on ...
1
vote
0answers
87 views

Strategies for dealing with Stale data and Fetching

I'm working with firestore and have encountered several situations where a document will contain stale data which is a duplicate of the source of truth located elsewhere in the database. For ...
1
vote
1answer
58 views

how simulate sequential consumers in Kafka

I have kafka consumer that get data from kafka and do some process if some circumstances meet and send it for another consumer to do it's job. How can i build something like this? One way I think ...
1
vote
0answers
74 views

separate table for one of the type or java enum?

I want to make budget tracking web-app, tracking all savings and expenses with corresponding dates. Later generating reports, summaries etc.. I came up this design Frequency Table - does it have to ...
1
vote
0answers
58 views

How can I offload customer's data needs onto the customer?

I write code for an Engineering application where engineers keep giving me various specs in a form of tabular data - for example, a table with product model names and many various parameters for each ...
1
vote
0answers
68 views

Java dashboard for machine data

I am currently working on a project for a production facility. The machines in the production are saving so called 'Telemetry Data', measurements like water pressure, plastic volume or temperature ...
1
vote
0answers
41 views

KNN Implementation in production for a retrieval arquitecture (CBIR)

We are going to start the implementation of a Content-based image retrieval (CBIR) that It will be integrated in our main product. We did tests and proof of concepts with different libraries that ...
1
vote
0answers
226 views

How to maintain table relationship across services in with a micro-service architecture

I have a scraper service that looks for media online. For every source type, there are two tables. A common source table and a source_type_media table that are related by source.id to ...
1
vote
0answers
194 views

Polling Postgres DB architecture

We are using Spring Integration DB polling which polls a table every second and executes some logic based on result. My questions are: What are some good practices we can implement to account for DB ...
1
vote
0answers
238 views

Are there any advantages of Cloud SQL Proxy over a DBaaS?

I'm currently migrating an app from Heroku to Google Cloud Platform (GKE and Cloud SQL). With Heroku, I've become used to the incredibly simple practice of defining a DATABASE_URL and letting their ...
1
vote
0answers
44 views

Where do I store the record sources for a form's controls?

I'm working on a Windows application that is used for tracking document review and audit information for my company. The back end is a SQL server database and most of the user tasks are basic CRUD ...
1
vote
0answers
172 views

Location specific caching for read-heavy data (server architecture)

This project will serve many duplicate requests with location-specific answers. I.e. 10,000 people in New York will all get the same server response (a list of businesses in New York), but one person ...
1
vote
0answers
369 views

Using a hardware load balancer for PostgreSQL / MySQL

I need to ensure high availability of my 3-tier architecture system. Currently I have the database mirrored by using MySQL Master-Master. All app boxes talk to the primary DB server In case of ...
1
vote
0answers
74 views

Where does search query logic go? Database or Service code?

I have a finished Chinese-English dictionary website as a personal project and it works pretty great. I'm using node/Express with node-dirty, which is a barebones file-based JSON database that loads ...
1
vote
0answers
47 views

Using fixtures to maintain static database assets

I have a pattern that's something like this: Create a database model UsagePlan which stores how many usage tokens a paid customer receives in exchange for its respective monthly fee. Through my ORM (...
1
vote
0answers
663 views

DB design for a scheduling system

I am currently designing a system to support our employee staffing process. I collect weekly preferences from employees, and feed that into a system for staffing. Availabilities can be defined as ...
1
vote
0answers
84 views

Creating an autoupdate for my program and its database

I've already an offline updater as a download but I want to make it automatically. It's a client/server software and a ms sql database. 3 parts have to be updated: update of a proprietary software (...