39
votes
How can I efficiently diff a CSV file against a database?
Since your goal is to produce a list of changes, not to change the stored records, the way to go is to simply export the database to text and wrangle it once to get into exactly the same CSV format. ...
- 107k
26
votes
Accepted
How to avoid violating the SRP in a class to manage caching?
One general-purpose mechanism for handling a broad range of cases where we want to add value without violating the Single Responsibility Principle is the Decorator Pattern.
This mechanism is suitable ...
- 31.9k
22
votes
Accepted
How can I efficiently diff a CSV file against a database?
Since the roundtrip seems to be the issue, you could:
either opt for a local solution, with the scaling issue you mentioned. (You could still try to split the task across several local nodes, each ...
- 72.7k
17
votes
Accepted
Caching authenticated requests for all users
I've been trying to address a similar issue.
My users need to be authenticated for every request they make.
I've been focusing on getting the users authenticated at least once by the backend app (...
- 286
17
votes
Accepted
Microservices - sharing data through distributed in-memory cache?
Just like using a single database for multiple services, the approach you describe causes a strong coupling between the services. For example, you can not change the data model of one service without ...
- 3,464
10
votes
Do application-level caches belong inside web servers?
As always, one must differentiate: what information is to be stored in the cache?
I always go with these simple rules:
Information every webserver instance can calculate for itself should go into an ...
- 4,538
9
votes
How can I efficiently diff a CSV file against a database?
There are already some good answers, but here's another possibility: if you could sort the CSV file by key and add a similar ORDER BY to the SQL query, you could compare the rows obtained from the ...
8
votes
Implementing a caching microservice by avoiding potential bottlenecks
My suggestion is that instead of worrying about whether to use an external or internal cache, your first concern should be that your booking-service does not care whether or not your are using an ...
- 5,221
8
votes
Where is network data stored while awaiting to be written to disk?
There would have to be quite a discrepancy in the age of the respective components, for a network adapter (and connected infrastructure) to exceed the performance of system RAM, so the premises of ...
- 6,443
7
votes
How to avoid violating the SRP in a class to manage caching?
I believe your class is doing one thing; it's a data cache with a timeout. LoadFluffies seems like a useless abstraction unless you call it from multiple places. I think it would be better to take the ...
- 171
7
votes
Accepted
Is cache an optional dependency?
PSR-6 recommendation regarding Caching Interface expresses a view I personally prefer:
While caching is often an important part of application performance,
it should never be a critical part of ...
- 186
7
votes
Possibility of stale data in cache-aside pattern
You are correctly pointing out a race condition.
Cache-aside, as described here and here, is an imperfect abstraction that is not appropriate for all data storage use cases.
Consistency. ...
- 33k
7
votes
Accepted
How is caching used within messaging apps?
I know that they store messages in cache... but wouldn't just a common database be enough?
So at time of writing, Whatsapp has... north of 1.5 billion users (https://www.statista.com/topics/2018/...
- 109k
6
votes
How to avoid violating the SRP in a class to manage caching?
Your class itself seems fine to me, but you're right in that LoadFluffies() does not exactly what the name advertises. One simple solution would be to change the name and move the explicit reloading ...
- 4,098
6
votes
Accepted
Storing objects whose size are run-time dependent contiguously in memory
How do I store objects, whose sizes are known only at runtime, contiguously in the memory? The sizes of the objects in one container are identical.
Allocate large chunks of memory.
Use placement new ...
- 1,956
6
votes
Accepted
What to call data that's not ready yet?
I think you're describing two different situations:
what do you call data that is not yet fresh, and has never been fresh?
Garbage. When you read a value before initialization happens, you read ...
- 99.6k
6
votes
Accepted
How to align on both word size and cache lines in x86
To understand how alignment affects things, let's look at a larger context.
First, as you note, 2600 bytes of UTF-8 (or any kind of data) will indeed take 2600 bytes.
If you allocate 2600 bytes from ...
- 33k
5
votes
Class design, responsibility granulation, efficiency and performance
You've possibly already seen some variant of Donald Knuth's "premature optimization" quote:
We should forget about small efficiencies, say about 97% of the time: premature optimization is the root ...
- 6,853
5
votes
Accepted
Why are the test values the same for FIFO and LRU?
It is not exactly a coincidence, it is a failure of your testing methodology to use the cache in such a way that would make the superiority of LRU over FIFO evident.
The superiority of LRU is evident ...
- 31.9k
5
votes
Why is CPU cache memory so fast?
There are a lot of good points raised in the other answers, but one factor appears to be missing: address decoding latency.
The following is a vast oversimplification of how memory address decoding ...
- 17.6k
5
votes
Why can I manually manage the GPU cache, but not the CPU cache?
This kind of memory management: telling the cpu (in advance) what content is frequently accessed, is really hard to do for a wide array of programming problems, where data structures involve pointers ...
- 33k
5
votes
Accepted
Caching data vs. caching output
Output caching has big benefits in applications that need to perform complex queries or calculations before displaying data, or that spend a significant part of their time rendering (turning the data ...
- 6,282
5
votes
Accepted
Using `304 Not Modified` and `If-Modified-Since` on a REST API as a cache strategy
If-Modified-Since works well for single resources where it is quick and easy to determine if the resource has been modified. For example, HTTP servers have this functionality baked in for static ...
- 45.6k
5
votes
Accepted
API design and storing custom queries from Db to cache
Caching database results may or may not be something you want to do, according to the situation. For example, if you want to retrieve live stock values to show the the user, it is obviously more ...
- 22.6k
5
votes
How to align on both word size and cache lines in x86
It seems your confusion comes from mixing up some architectural levels.
You have your processor architecture that may be 64 bits, making it "easy" to work on chunks that align with 64 bit boundaries. ...
- 17.9k
4
votes
Are there CPUs that perform this possible L1 cache write optimization?
Writing to L1 cache is a very, very time critical operation.
Writing the exact same data back seems to be rather rare. An optimisation that speeds things up in this particular case isn't going to ...
- 40.9k
4
votes
How to avoid violating the SRP in a class to manage caching?
Your instincts are correct. Your class, small though it may be, is doing too much. You should separate the timed refresh caching logic into a completely generic class. Then create a specific instance ...
- 33.6k
4
votes
How to implement "viewed: N times" functionality for an article?
I would suggest updating the counter on each view. As @amon suggested - that hard part is determining WHEN to update the counter.
An old adage - no premature optimization before its time. ...
- 141
4
votes
Caching mechanism for REST APIs
For a REST service, you usually want to use the available caching mechanisms of the web and the HTTP protocol. That means using caching directives or entity tags in the responses.
In this context, ...
- 3,600
Only top scored, non community-wiki answers of a minimum length are eligible