13
votes
Is it worth logging HTTP requests when they enter an API server?
What is the goal of your logging? Logging request, response, and user info is perfectly valid if your goal is to build a profile of your users. Google certainly does.
If all you want to do is debug ...
- 115k
11
votes
Why is elastic search popular?
MongoDB is a database. Elasticsearch is a search engine. Since their aims are different, they have different priorities. MongoDB is focused on storing data consistently with good performance and to ...
- 9,217
6
votes
Accepted
Logging architecture in microservices
I have a few recommendations based on my experience.
I would call user action logs user audit logs to differentiate them from application logs.
ELK should be a fine place to store both kinds of logs.
...
- 3,123
6
votes
How to handle UI updates dependent on slow API responses
You don't have to block the complete UI before the response comes back. You only have to disable the parts of the UI which allow to make another API call before the first one is processed completely (...
- 214k
5
votes
Why would I use ElasticSearch if I already use a graph database?
Both of these database has their specific need to solve specific problem at certain level of application requirement.
Although we have not used Graph Database. But we are using elasticsearch with ...
- 151
4
votes
Is it worth logging HTTP requests when they enter an API server?
Logging all API requests can be extremely important for security and compliance. A lot of intrusions can be detected or investigated later on based just on log analysis. Without those, it's like going ...
- 141
3
votes
eCommerce category path - names vs IDs
Both names and IDs create a series of difficulties. You already listed a few ones in your question, such as the removal of a parent category. Another major issue is that categories are a completely ...
- 136k
3
votes
Architecting a universal search for a product with microservices
I would somewhat disagree with @Ewan. While yes, you can wind up with a big blob of unstructured data, that’s only an inevitable consequence if you don’t work to prevent it.
In counterpoint, I’d ...
- 3,327
2
votes
Design point for product search and auto-complete using Elastic search?
As you've written this question, it describes the standard usage for a search engine: you index the fields that you want to search by, then run queries against those fields.
A search engine, however, ...
- 5,270
2
votes
Architecting a universal search for a product with microservices
I Would avoid a 'Central Service' you can end up with a big blob of unstructured data.
Presumably once the user has found something in there, you will want to do something with the result and this ...
- 79.8k
2
votes
Elastic search vs chatbot?
The common term for what you're describing is "expert system", or for a particular style a "software wizard". That you're talking about implementing it via text interface is just an implementation ...
- 334
2
votes
Accepted
Elasticsearch and PostgreSQL combination
Think of Elasticsearch as an eventually inconsistent index on top of your DBMS. The index is built once for a specific state of the database, but when the content of the DB changes, one will have to ...
- 214k
2
votes
How to architect an in-app search solution that accounts for access to data?
relational model
This is a classic RDBMS JOIN.
So ditch ES and use some proper backend data store, perhaps Postgres.
It will be performant.
RBAC
Creating a relation that models per-user fine grained ...
- 7,605
2
votes
REST API Design: Should we have separate endpoints for same resource if the system uses both ElasticSearch and MySQL?
REST doesn't care. The fact that you are using databases rather than the file system, and two databases rather than one, is an implementation detail that is hidden behind the web interface.
As long ...
- 33.7k
2
votes
Accepted
Do I have to use Filebeat to parse logs with logstash?
No, you don't have to use Filebeat if you choose to use the file option.
You can easily test this using the following conf file:
input {
file {
path => "/var/log/debug.log"
...
- 136
2
votes
Is it worth logging HTTP requests when they enter an API server?
Best practice is to log whatever you need. You should know your application well enough to know what information is required to troubleshoot issues. If you don't, then spend more time thinking about ...
- 3,123
1
vote
Finding related items in OpenSearch between 2 datasets
You are describing a need for a
recommender system.
There is a rich literature on this topic for you to consult,
with many implementations.
You didn't mention data volumes,
nor how much consumer ...
- 7,605
1
vote
What is the advantage of Kafka connector to MongoDB over using MongoDB change streams?
You have a classic trade off between a simpler narrower-purpose solution and a more complex, but more general purpose, solution. Change streams work for streaming changes from MongoDB, but what if you ...
- 148k
1
vote
What is the advantage of Kafka connector to MongoDB over using MongoDB change streams?
Well if I understand the situation sufficiently, then I would say that the solution is neither one, but rather using them BOTH together.
This is a solution which is very commonly used all across the ...
- 150
1
vote
Elasticsearch and RDMBS combination
As you have observed, relational database design is largely about reducing the number of times data is represented through normalization. ElasticSearch is, by and large, all about denormalization and ...
- 6,467
1
vote
Is AWS Elasticsearch is completely managed service or is it just a Elasticsearch installation on set of ec2 instance
It's a managed service operated by AWS.
That being said I'd look at https://cloud.elastic.co/ which can run on AWS, GCP and Azure. 14 days for free. It's managed by elastic, creator of elasticsearch.
...
- 111
1
vote
Is it worth logging HTTP requests when they enter an API server?
It depends if you can afford the extra log then why not. Especially now that we have central logging solutions etc... Sometimes cases arise when your like I wish I had the request/response bodies.
One ...
- 111
1
vote
Accepted
Designing a Social Network with Graph Databases
If you've already decided (and I'd agree) to store the text in an another database and a key reference to it inside the graph edge, there's no need to store a copy of text there as well, that would ...
- 507
1
vote
Some Guidance on Parent-Child Relationships in Elasticsearch
Method 2 Sounds Best
It's simplest and seems most suited to the situation you describe.
It has been over a year since I worked on Elasticsearch. This is what I recall.
Why? Method 1 vs. Method 2
...
- 3,501
1
vote
Decouple service complex query building from datastore
I also dont think that you need to decouple the two if your only concern is that ElasticSearch is updating the interface. If you have a decoupled layer then it should be more or less the same effort ...
- 170
1
vote
Problems with evaluating results of search engine by comparison
Comparison with an oracle is always a good idea, when available. Comparison with a principle competitor is also a good idea.
Your metric for comparison doesn't appear well thought out (what if the ...
- 2,955
1
vote
Accepted
Design the cache and search feature like hotel booking?
Seems like you want to have a search engine. Have you already considered using elasticsearch, instead of implementing such a complex piece of software by yourself?
- 203
1
vote
Data layer Abstraction from Business layer for search
Well, I would advocate performance that benefits client over the choice of developer ease (after all client is the one who is paying).
If you really have these two options then go with
Search-> BL ->...
1
vote
Architecting a universal search for a product with microservices
We are building a new product in real estate space and the end users of this product are not so tech savvy. To have better user experience with our product, we want our users to find relevant things ...
- 3,995
1
vote
ElasticSearch vs SQL Server full text index for small datasets?
Keep it how it is. If it ain't broke, don't fix it.
Generally, "simplicity" is very important. To illustrate this reasoning, let's take it to an extreme:
You have 30k records
You have 30 ...
- 880
Only top scored, non community-wiki answers of a minimum length are eligible