What's the potential use for capped collections? (besides logging)
Capped collections are useful for niche use cases where:
- You want to limit the amount of data retained in a FIFO (First In, First Out) order
- You want to guarantee data is stored in insertion order
- You do not have updates that cause documents to grow in size
- You do not need to delete documents manually
- You may want to use a tailable cursor to trigger some activity based on inserts into a collection
Several of these characteristics (for example, not allowing documents to grow in size) may sound unusually restrictive, but there were a few significant motivations for the original design: enable the highest possible insert rates for the original MMAP storage engine and ensure that documents in capped collections have the same size when replicated. The MMAP storage engine preallocates files for the entire capped collection to minimize storage fragmentation and avoid concerns about running out of disk space before a collection reaches its maximum capped size.
MongoDB 3.0 added a storage engine API, and in MongoDB 3.2 the default storage engine for new deployments changed from MMAPv1 to WiredTiger. All storage engines currently support capped collections as part of the MongoDB API but the internals and efficiency of implementations will vary as compared to the original MMAP storage engine.
Since capped collections are effectively large fixed-size circular buffers, the most obvious use cases are related to logging. Typically a certain amount of "recent" data needs to be kept and aged out in a FIFO manner rather than growing indefinitely.
MongoDB relies on capped collections for a number of internal use cases such as:
local.startup_log, a 10MB capped collection which captures some diagnostic/environment information about the local
mongod instance on start up.
- The replication oplog (operation log), which each replica set member uses to record an idempotent list of all write operations applied locally. The default oplog size is based on % of free disk space, but is typically on the order of up to 50 GB.
- The sharded cluster
config.changelog, which is a 10MB capped collection of recent sharded cluster balancing activity.
You could conceivably use capped collections for cases where you want data to age out automatically (i.e. a FIFO cache or queue), but a normal collection with a Time-To-Live (TTL) index generally provides much more flexibility as well as control over document expiry.
Capped collections cannot be sharded, but they can be replicated. In case of network partition and after rejoin, how is capped collection synchronized/merged?
Sharding partitions a collection across multiple servers and is contrary to several of the capped collection design goals listed above. Replication is an asynchronous (but ordered) copy of write operations to all members of a replica set.
In a replica set environment writes to capped collections propagate using the same replication mechanism as all other writes. Data on secondaries will be eventually consistent depending on replication lag, but still preserve the insertion order and other capped collection properties. Replica sets only allow a single primary, so in the event of a network partition a primary will only be elected for a partition with a majority of healthy nodes. Any isolated members on the minority side of a partition will remain as readonly secondaries until they are able to rejoin the replica set and attempt to resume replication.