DB write latency in BadgerDB high when the size of the data increases exponentially

[abbas-unmarshal (Abbas)]

@iluminae

You may wish to share here the attempted configurations and results at each configuration.

Right now it is on default configuration. Taking an average of 10sec to write a single block

The below configuration was taking 5sec to write a single block without any existing data, so we did not go ahead and use these configuration.

opts.MemTableSize = 1 << 20
opts.BaseTableSize = 1 << 10
opts.NumMemtables = 3
opts.NumLevelZeroTables = 3
opts.NumLevelZeroTablesStall = 6
opts.BlockCacheSize = 10 << 20
opts.ValueThreshold = 1 << 10

We also tried this configuration which reduced our memory usage on the cost of write speed, but our priority was write speed so we did not go ahead with these as-well. Not a major change, just having lesser tables in memory

opts.MemTableSize = 1 << 20
opts.BaseTableSize = 1 << 20
opts.NumCompactors = 2
opts.NumMemtables = 1
opts.NumLevelZeroTables = 1
opts.NumLevelZeroTablesStall = 2
opts.BlockCacheSize = 10 << 20
opts.ValueThreshold = 1 << 10

@iluminae

The disks that back badger are extremely important to be low-latency, but if you had a similar test with LevelDB (very similar read/write pattern) on the same disks I would expect relatively similar access performance in go.

We are using GP3 SSD on AWD with and IOPS of 3000 as disks for both leveldb and badgerdb.

@iluminae

edit: also the equivalent leveldb that succeeded in your tests would be interesting. EG: compression settings

The configurations we use for Leveldb.

BlockCacheCapacity:     128 * opt.MiB,
WriteBuffer:            1024 * opt.MiB,
DisableSeeksCompaction: true,
OpenFilesCacheCapacity: 1024,