How to exchange 1T data between multiple processes?


  1. Data < 1TB. Memory > 1T.
  2. Multiple processes (different languages) on the same machine or different machines over network
  3. Data should be available for 1 hour for fast read
  4. Guarantee completeness of data.

I am thinking using JSON or Redis or 0MQ as the solution. Are there other solutions? What are the pros and cons?

  • 1
    see What is the problem with "Pros and Cons"?
    – gnat
    Dec 31, 2019 at 7:19
  • 4
    Try any approach. Hit the wall. Understand why you hit the wall, and either optimize or change approach. Rinse and repeat until satisfied. Dec 31, 2019 at 7:22
  • What kind of data? What kind of application? What operating system? Dec 31, 2019 at 8:48
  • What do you mean by "1T"? Do you mean "1TB"?
    – Doc Brown
    Dec 31, 2019 at 16:20
  • This question makes absolutely no sense to me - there is way too much lacking information to give a meaningful answer.
    – Doc Brown
    Dec 31, 2019 at 16:31

1 Answer 1


Read much more about serialization (e.g. this) and persistence and distributed data store and big data and data science. There are libraries, formats like XDR, tools like SWIG which could help. Consider keeping the data in some centralized database server (PostGresQL, MongoDb, Cassandra ...). Read about the challenge of Distributed Database Systems, ACID and the CAP theorem and Rice's theorem and π calculus. Read about middleware for distributed applications.

If you use any relational database, put efforts on defining well your database schema and database indexes and be aware of database normalization.

If the processes involved are on the same computer, read about MPI, inter-process communication, shared memory. So read much more about operating systems.

On Linux, see also socket(7), fifo(7), shm_overview(7), mq_overview(7)

Textual formats and text based protocols like JSON, HJSON, XML, s-expressions could be helpful (easier to debug) but requires CPU time to be parsed. So read about parsing. Given that CPU are a lot faster than disks (see http://norvig.com/21-days.html ...) there are situations where compressing the data (using lossless compression à la zlib) before storing it on disk is worthwhile.

The devil is in the details. Put efforts on documentation of your software architecture and data format. The data (and its collection) usually costs more than the software processing it.

Your question is really too broad. I tried to give some hints.

PS. You could get a PhD in solving your problem efficiently. Look into ACM conferences (e.g. this one) and ACM papers related to it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.