New answers tagged database
0
You should do both (and, if there's a web-based User Interface involved, you should be limiting the length of the input field there as well!)
Yes, it means you have the length of this field defined in multiple places but that's no different to having field names in SQL in the Application code. If those become "inconsistent due to programmer oversight&...
0
This is not an answer to the literal question you posed, but based on the (supposedly) underlying requirement.
You typically declare a varchar length because you know that longer strings are invalid for that field, because of some business logic / domain analysis.
In my experience, all cases with a limited string length also had some syntax requirements as ...
1
It depends on the level of abstraction on the diagram.
A component on a component diagram usually represents more than just a single class, but that doesn't mean that it can't represent a class. A component identified as a "database component" could be the class or classes used to represent a connection to a database. It could also be the database ...
-2
UML is an object oriented modeling language so components do represent classes but they will be course level descriptions lacking detail. The point is to outline the big chunks and communication routes between them, to show what entity talks to what other entities and to get a high level overview of the entire system.
So it does represent the entire database ...
2
Your code should be always prepared for getting an error from the database during an insert operation, and to display a useful error message in this case. Hence "either - or" is the wrong question: you should ask if it makes sense to validate the string length prior to inserting additionally.
However, when you insert a lot of columns in one ...
1
All you need is another field with the server id. Then your search for duplicate entries should include that this entry needs to have the current server id, too.
I'm pretty sure you can do that with any database (Sql or NoSql) or any other storage/query mechanism.
0
My question is: where in this design is scalability defined? Or rather, if we scale up the volume of data that goes in and out, which part of this design would start giving issues?
Scalability is usually separated into horizontal or vertical scaling.
"Horizontal scaling" for a data pipeline usually refers to the ability of partitioning the input ...
0
Based on your comment on DFord's answer:
I was thinking of splitting records in chunks of X size if the total amount exceeds a certain threshold, thenhanding the chunks to different threads
It seems you're running 1 process of your app, on-premise, not being able to benefit from the cloud FaaS approach (which I think it would be the best for cost-effective ...
1
Its hard to point at one part of the data pipeline as the possible bottle neck that will prevent it from scaling.
I am not sure if this is already being done, but running the Python application as a FaaS where it can scale up and down based on load could help. Or having multiple instances of the python application running to process the data more quickly.
...
2
So essentially you've run into performance issues, and decided to go lower down in abstraction levels. This is a sensible approach although sometimes it's not the most long term approach; but it certainly tackles the problem.
I'd like to respond to something about your concerns and beliefs this is bad:
The current version uses Hibernate's native query ...
1
Before digging in some technical details I'd like to share some higher level considerations:
The amount of times you need to build your database for a database requirement is low, and it's better double check whether or not an on the shelf solution would be better suited; or a solution built on top of an on the shelf solution. Too many times, rewriting a ...
1
Begin db transaction
Update the db, setting a status or a flag that means something like "processing".
Commit transaction
Perform the API call.
If the API call succeed, the process still in control, begin transaction, do your db updates, also update the status/flag to "done", commit transaction. All good.
If the API call fails, or the ...
1
Microservices aim to be independently deployable. In this regard, it all depends how interleaved the documents and users are.
The user example is moreover a delicate example, since users may also be related to authentication and authorizations and might therefore be used in your scenario for different purposes:
From a very general point of view, the user id ...
1
You need to analyse this from the point of view of use cases, not just "API calls".
I have an api call that gets all the documents that are shared for a user, each document belongs to a user so if i fetch a thousand documents i have to make a thousand additional call to get the user data.
Ok, but what is the use case where this API call is ...
1
I you really want to test the details of DB operations in your code, the method is the same as always: you mock the component that receives the operations and assert that the received operations satisfy certain constraints.
For instance, you might assert that all the operations are wrapped into transaction boundaries (explicitly or implicitly). (For bonus ...
2
No, it’s not ok. Someone claimed:
security is like onions: the more layers you peal, the more you cry.
This refers to the concept of layered security. There are many ways your system could be attacked: from the outside by compromising your public service; but also from the inside via another service or system: this is called a “lateral move” by the ...
4
The security argument against using DDLs is neither completely wrong nor completely right.
If a web service is compromised, and an attacker manages to issue arbitrary select, insert, update or delete statements, the damage they can cause is not really different from the damage they can cause by a drop table or modify table statement. Ok, when you system ...
3
Are you absolutely, completely, 100% sure that there are no vulnerabilities in your code and all the libraries you're using to access the database that might allow SQL injection or the like?
If you are sure, then there's no benefit. If you're not, then the benefit is obvious.
(PS: you're not 100% sure)
0
You have a classic trade off between a simpler narrower-purpose solution and a more complex, but more general purpose, solution. Change streams work for streaming changes from MongoDB, but what if you have sources other than MongoDB in the future? Also, I don't know how change streams scale, but kafka topics are designed from the ground up to partition for ...
0
This sounds like a machine learning problem.
Take you existing dataset and look for groups via classification
This is going to be tricky with your object having many dimensions. But once done you can now look at a new object and classify it based on its data + your model. Rather than having to look at all other objects.
Essentially this is the same as ...
2
In general, this can be a pretty hard problem, especially when you want to implement it using traditional databases. There are special cases which are easy to solve, you have to check by yourself if your requirements will allow to exploit such cases, or if you need a fully general solution. It makes also a pretty huge difference if you want to check string ...
Top 50 recent answers are included