DB write latency in BadgerDB high when the size of the data increases exponentially

Issues Badger
6

@abbas-unmarshal Is there a reason for using such small sizes for memtables? Your writes would become slow if badger cannot push data to lower levels.
I have a couple of suggestions to increase the write speed

  1. Set memtable size back to 64 MB (the default value)
  2. Use more compactors. More compactors mean badger can move data at a faster speed to lower levels and it won’t block writes.
  3. Use BatchWriter instead of the Txn. Your write speed would also depend on how you’re doing writes. If you commit for every small entry, your write speed would be lower. The batch writer is much much faster.
  4. Disable compression/encryption and disable the cache. This is very important. Compression/encryption are CPU-intensive processes and they will slow down your writes.

I am happy to help you optimize for write speed if you can help me with a sample program that replicates your usecase. There might be other options that can help you speed up.

1 Like