Is there a FIFO mode in Badger? That is the ability to
a) do not do compactions
b) client can directly read the oldest SST
c) delete it when done?
Hi @joriwils , I am interested to know more about this use case. Can you please link any external references.
Hi Naman,
RocksDb describes it here: https://github.com/facebook/rocksdb/wiki/FIFO-compaction-style
The use case is a searchable log. That is the client adds Key to the LSM when it the corresponding Value enters the cache. Each key might get a Log Sequence Number (LSN). Then after some time the KV pair is discarded. In that way “log wrap” is emulated.
The FIFO mode has two benefits: there is no compaction cost and no deletion cost. KV pairs are deleted by deleting the oldest SST. For example the master application manages the log content by LSN. Once it knows that all KV pairs with an LSN less than the lowest LSN in an SST have been discarded/passed on, it can delete the SST. Obviously, the FIFO mode has high search costs if the number of SSTs is high. However, if the number of SSTs can be kept low or low search cost is not a major goal, then FIFO compaction works.
I suspect Facebook uses it in their LogDevice: LogDevice: a distributed data store for logs - Facebook Engineering