I am building a three-tier architecture application that receives compressed (zlib) text records from a data store, decompresses the records into structs, and then serves the records to clients over the network, which run SQL-style queries on the records.
The issue I'm running into is I need this application to be able to serve potentially hundreds of gigabytes of uncompressed text records, far too much to be able to store in memory.
The tricky part here is that these records are nearly completely updated every hour, so my solution needs to be able to fetch data quickly enough so that hourly downtime does not become a problem.
Previous versions of the server simply read from the data store files directly, which led to reliability problems and race conditions with the data store itself. Now, I've reworked the server to receive transmissions from the data store, but this comes with the issue of where to store all that data while it's being served.
What are some good techniques for minimizing memory usage on the server while also keeping query execution time relatively low?