Skip to main content
39 votes

How can I efficiently diff a CSV file against a database?

Since your goal is to produce a list of changes, not to change the stored records, the way to go is to simply export the database to text and wrangle it once to get into exactly the same CSV format. ...
Kilian Foth's user avatar
22 votes
Accepted

How can I efficiently diff a CSV file against a database?

Since the roundtrip seems to be the issue, you could: either opt for a local solution, with the scaling issue you mentioned. (You could still try to split the task across several local nodes, each ...
Christophe's user avatar
  • 78.7k
18 votes
Accepted

Caching authenticated requests for all users

I've been trying to address a similar issue. My users need to be authenticated for every request they make. I've been focusing on getting the users authenticated at least once by the backend app (...
mbarthelemy's user avatar
17 votes
Accepted

Microservices - sharing data through distributed in-memory cache?

Just like using a single database for multiple services, the approach you describe causes a strong coupling between the services. For example, you can not change the data model of one service without ...
Michał Kosmulski's user avatar
9 votes

How can I efficiently diff a CSV file against a database?

There are already some good answers, but here's another possibility: if you could sort the CSV file by key and add a similar ORDER BY to the SQL query, you could compare the rows obtained from the ...
Daniel Koszta's user avatar
8 votes

Implementing a caching microservice by avoiding potential bottlenecks

My suggestion is that instead of worrying about whether to use an external or internal cache, your first concern should be that your booking-service does not care whether or not your are using an ...
TheCatWhisperer's user avatar
8 votes

Where is network data stored while awaiting to be written to disk?

There would have to be quite a discrepancy in the age of the respective components, for a network adapter (and connected infrastructure) to exceed the performance of system RAM, so the premises of ...
Steve's user avatar
  • 9,272
7 votes

Possibility of stale data in cache-aside pattern

You are correctly pointing out a race condition. Cache-aside, as described here and here, is an imperfect abstraction that is not appropriate for all data storage use cases. Consistency. ...
Erik Eidt's user avatar
  • 34.2k
7 votes
Accepted

Is cache an optional dependency?

PSR-6 recommendation regarding Caching Interface expresses a view I personally prefer: While caching is often an important part of application performance, it should never be a critical part of ...
Небојша Камбер's user avatar
7 votes
Accepted

How is caching used within messaging apps?

I know that they store messages in cache... but wouldn't just a common database be enough? So at time of writing, Whatsapp has... north of 1.5 billion users (https://www.statista.com/topics/2018/...
Telastyn's user avatar
  • 109k
6 votes
Accepted

Storing objects whose size are run-time dependent contiguously in memory

How do I store objects, whose sizes are known only at runtime, contiguously in the memory? The sizes of the objects in one container are identical. Allocate large chunks of memory. Use placement new ...
R Sahu's user avatar
  • 1,986
6 votes

Why can I manually manage the GPU cache, but not the CPU cache?

This kind of memory management: telling the cpu (in advance) what content is frequently accessed, is really hard to do for a wide array of programming problems, where data structures involve pointers ...
Erik Eidt's user avatar
  • 34.2k
6 votes
Accepted

What to call data that's not ready yet?

I think you're describing two different situations: what do you call data that is not yet fresh, and has never been fresh? Garbage. When you read a value before initialization happens, you read ...
candied_orange's user avatar
6 votes
Accepted

How to align on both word size and cache lines in x86

To understand how alignment affects things, let's look at a larger context. First, as you note, 2600 bytes of UTF-8 (or any kind of data) will indeed take 2600 bytes. If you allocate 2600 bytes from ...
Erik Eidt's user avatar
  • 34.2k
6 votes
Accepted

Can caching increase p90 response time

This is the trouble with caching things. The cache is fast only if this is the second call and we hit the cache. Stray anywhere outside this and a cache solves nothing. In your case, a cache doesn't ...
Greg Burghardt's user avatar
5 votes
Accepted

Caching data vs. caching output

Output caching has big benefits in applications that need to perform complex queries or calculations before displaying data, or that spend a significant part of their time rendering (turning the data ...
JDT's user avatar
  • 6,410
5 votes
Accepted

Using `304 Not Modified` and `If-Modified-Since` on a REST API as a cache strategy

If-Modified-Since works well for single resources where it is quick and easy to determine if the resource has been modified. For example, HTTP servers have this functionality baked in for static ...
Berin Loritsch's user avatar
5 votes
Accepted

API design and storing custom queries from Db to cache

Caching database results may or may not be something you want to do, according to the situation. For example, if you want to retrieve live stock values to show the the user, it is obviously more ...
Neil's user avatar
  • 22.8k
5 votes

How to align on both word size and cache lines in x86

It seems your confusion comes from mixing up some architectural levels. You have your processor architecture that may be 64 bits, making it "easy" to work on chunks that align with 64 bit boundaries. ...
Martin Maat's user avatar
  • 18.5k
5 votes

ASP.NET MVC Caching of ViewModels

It doesn't make sense to cache the ViewModel ViewModel + View = HTML. So if your view model hasn't changed then the HTML hasn't changed. You can just return the cached page. Cache the models, because ...
Ewan's user avatar
  • 77.5k
4 votes

How to implement "viewed: N times" functionality for an article?

I would suggest updating the counter on each view. As @amon suggested - that hard part is determining WHEN to update the counter. An old adage - no premature optimization before its time. ...
ripvlan's user avatar
  • 141
4 votes

Caching mechanism for REST APIs

For a REST service, you usually want to use the available caching mechanisms of the web and the HTTP protocol. That means using caching directives or entity tags in the responses. In this context, ...
Bogdan's user avatar
  • 3,620
4 votes

What to call data that's not ready yet?

If it's never been saved to the database, I'd call it "unsaved" or "uncommitted", but for clarity, I'd be a little more verbose in things like documentation and error messages, and refer to them as "...
bigblind's user avatar
  • 1,425
4 votes

Design for 'inbox' with 'unread' feature

Its a big topic but here are some general recommendations Redis Cache should not be part of your solution design. It's something you put in to deal with load, not to achieve a solution. Get your ...
Ewan's user avatar
  • 77.5k
4 votes
Accepted

Best way to reduce database writes?

Assuming that you really need this counter, does it really have to be constantly updated every time a user sends a message? Could you use a more "granular" counter which increments every ...
Alex D's user avatar
  • 1,308
4 votes
Accepted

Is it possible for the CPU to read or write to the cache, while I/O device is reading main memory?

Without miss, no problem. With a miss that can be filled from L2 or L3 cache, no problem. High end CPUs can have two paths to separate RAM chips, so they could run simultaneously. CPU and external ...
gnasher729's user avatar
4 votes

Best way to store quite large JSON objects?

You have an interesting scenario with lots of solution that might be appropriate. You raise a good point that the amount of data might be an issue. As a worst case estimate, you might have 0.5MB per ...
amon's user avatar
  • 135k
4 votes

How does Java and other managed languages achieve any performance, if everything is allocated at random places of the heap?

After thinking on it for a while, I have no idea how developers in banks, some HFT companies (!!!), stock markets and other businesses that require high performance, manage to write fast (not "...
candied_orange's user avatar
3 votes

Using `304 Not Modified` and `If-Modified-Since` on a REST API as a cache strategy

The drawback of your approach is that it doesn't cover many of the caching scenarios. Every element seem to be cached forever (at least for a very long time), and when the client runs out of memory, ...
Arseni Mourzenko's user avatar
3 votes

Implementing a caching microservice by avoiding potential bottlenecks

I only see three good reasons why the caching should be extracted: Multiple internal services interact with the same external service. If all your services would share a cache, this could be a net ...
amon's user avatar
  • 135k

Only top scored, non community-wiki answers of a minimum length are eligible