We’re rewarding the question askers & reputations are being recalculated! Read more.

Hot answers tagged

26

We use both Redis and Zookeeper at work so this is from first hand experience Redis is fast; really, really fast. It is also immediately consistent, so it's good for fast moving data sets. The downside is that, running on one server, if it fails then you lose write access until another server takes it's place. Replacing the server is a manual operation ...


22

I've run into some difficulties thus far I'd like to document here. How do you handle reconnect logic? This is a hard problem and an especially hard problem in designing and implementing a message queue. Messages must be able to queue up somewhere when consumers are offline, so a simple pub-sub is not strong enough, and consumers need to reconnect in a ...


15

Redis is an in-memory database because it keeps the whole data set in memory, and answers all queries from memory. Because RAM is faster than disks, this means Redis always has very fast reads. The drawback is that the maximum size of the data set is limited by the available RAM. Redis has various options to save the data to permanent storage. This ...


8

Just like using a single database for multiple services, the approach you describe causes a strong coupling between the services. For example, you can not change the data model of one service without having to accordingly change the model in others. This means that both development and deployment are coupled and your microservices in reality become a ...


5

If you want to use Redis for a message queue in Node.js and you don't mind using a module for that then you may try RSMQ - the Redis Simple Message Queue for Node. It was not available at the time this question was asked but today it is a viable option. If you want to actually implement the queue yourself as you stated in your question then you may want to ...


5

If we consider the cache to be orthogonal to the architecture (and it's), the first pic is ok. For the same reason that we don't deploy one security service, one API gateway, one message broker or one service locator per POD we don't have to deploy one Cache (in replica-set) per POD.1 Be aware of premature optimizations Caches are meant to solve specific ...


4

Even fetching all details for just one hotel may results in a JOIN query from at least four tables, and scanning over all hotels records. A four-join query is absolutely trivial if you have the appropriate indexes for all joins. The second part of this question is far more troubling. Why the scan over all records? Is is because of missing indexes? or ...


4

Interesting challenge (!) because dates are sparse. Here's a thought: if you can find a way to represent dates as numbers (minutes past the year 2000?), then you could try a Sorted Set. The set would be keyed on the user ID. The score would be the numeric date representation. The value would be the log entry. Then you can use ZRANGEBYSCORE to get log ...


4

I think that your search results can greatly improve through a number of techniques or database design approaches that will improve performance in your typical RDBMS. I suggest looking into and possibly prototyping the following improvements to see if they help you in performance testing first before you commit to an entirely new database technology that ...


4

Separate services should use separate REDIS instances. Reason: Bad usage of REDIS by another service, causing REDIS outage, should not impact your application. Only queues which are used for inter-service communication should be shared.


3

You can attack this issue from a couple of angles: You can choose an eviction policy that will not evict those keys, e.g, eviction policy volatile-lru and set those keys without expiration. (more on eviction policies) The issue here is that you might get OOM if there are no keys that can evict. The second approach is to set an eviction listener, you can use ...


3

Yes, he meant logging. Redis is actually classical for logging - it's volatile data; writes should be extremely fast in order to not create an unnecessary load on the system/database; you are usually interested in only the last logs so you can easily run: LPUSH log error_message LTRIM log 0 1000 and always keep only the latest logs. Of course, this example ...


3

I am not familiar with Redis, however, some other key/value store databases have the following problems: Updates are not instantly visible immediately. SQL joins are to be performed by application. Some SQL features such as Distinct, Group By are to be performed by application. No stored procedures, triggers, etc. No FKs (2) and (3) above are particulary ...


3

Cache can be used and will help here as many clients for events which will rarely change. Even if they change: updating a key, or deleting and adding it back lazily, works fine. For exactly how you should do it - experiment. Plunge in with the only caveat : you will learn and change. Embrace that idea and run with it. I have seen code that stores same ...


3

Oh, the joys of distributed systems. There are two general approaches to this problem: You can keep using this caching strategy and accept that service A will usually get outdated data. This is just a performance vs consistency tradeoff. You can tune this tradeoff as necessary, for example by expiring cache entries after an acceptably short period of time. ...


3

Can you prevent that one service writes into the namespace of another service? If not, sharing one Redis instance breaks service separation. And soon Redis will be used as a way to sync data. Not to mention the risk that one service accidentally manipulates another service's data. If you can enforce the separate namespaces via Redis, i.e. different access ...


3

After further research I am satisfied with an answer so I will answer my own post. I see that Microsoft recommends using a prepended-tenant (client ID in this case) in this particular article: https://docs.microsoft.com/en-us/azure/architecture/multitenant-identity/token-cache Furthermore, I see that the framework I'm using has specific options & code ...


2

What would be stopping my website users from opening firebug or chrome's inspector and making changes to the database? If this is possible, you've got some serious redesigning to do. All your authentication should be on the server end of things. Your node server should be validating the request before it takes any action. End users fiddling with ...


2

I'm not sure how much redis will help here -- it can do some fancy tricks for sure, but you probably need to work out getting this thing built and running and have some baseline performance before sticking in a caching layer. Personally I would start with just using MongoDb's map reduce functions and take it from there.


2

Why either/or? I've worked very successfully with a hybrid approach, using a relational db (SQL Server, but pick your favourite) to hold data that needs a relational structure - most of this is IDs linking all the various domain objects, very little textual data and certainly no blobs - and a nosql db (Dynamo) to hold large relatively unstructured data, ...


2

NoSQL is generally not very good with relational data. NoSQL is often great for non-relational but structured data like documents or time series. Your "one to many" relationships may look quite like a document: e.g a "hotel" document may carry all its images, room info, etc stored together and fetched with one operation. On the other hand, if you see a ...


2

If your service is hosted on a single machine, chances are that in-memory caching will have a better performance and will be easier to implement. On the other hand, there are three situations where local caching is not enough: As soon as your service starts to be hosted on multiple servers, which is the case for most services hosted in production, you may ...


2

TCP is not a great protocol for this kind of thing. I would suggest a middle layer. A single TCP listener that receives all messages and puts them on a queue, where "a queue" means any queuing system that allows distribution of messages across multiple subscribers. RabbitMQ is pretty easy to implement in .NET and very flexible for your needs. Work Queues ...


2

I think you're overthinking things. You can use a traditional SQL-based RDBMS. It may or may not be fast enough (although I suspect you're worrying prematurely about optimization), but the only way to tell would be to try it. Just make sure that you write your code that interacts with the storage system in an abstract-enough way that it is simple to ...


2

If it only has a lifetime of 5 to 10 minutes, and there isn't a whole lot of it, then store it in memory. If those things are not true, any key/value store or relational database will do.


2

Spark programs are just java, scala or python code, so they can write data to all the same places any program can write them. In fact, spark does not actually do anything unless you write the end result somewhere with an output operation. If the end result of a spark job is small, it can be written to a relational database or a web service or something of ...


2

You don't provide enough information bout your system to recommend a solution. However, using a (two!) databases as a queue is less good than using a queue as a queue. In addition I would say you only need queues where you have 'long running processes' which are triggered faster than they can complete. I dont see where in your app you have this? It seems ...


2

I think the channel approach is preferred over a group identifier. Clients should only subscribe to messages that are actually important to them. While this may lead to a lot of Redis channels and connections, the alternative would create very chatty subscriptions where messages are being sent to subscribers but immediately discarded because they don't have ...


2

The DB will probably have performance issue since the capturing speed can be very high and viewer will constantly pulling data from it. Not to mention that you'll need an automated process to clean up the database. I think that a better solution would use a streaming log such as Kafka. The idea of such a log is that you have multiple brokers, each writing ...


2

I don't understand your data model in sufficient design to give you concrete advice, but can provide a few pointers about combining databases. Using multiple databases is perfectly fine because you can play to their individual strengths. Yet there are also a number of drawbacks: By combining multiple databases and other tools, you are introducing extra ...


Only top voted, non community-wiki answers of a minimum length are eligible