Memtable memory usage causes OOM

ahsan · August 13, 2020, 8:42pm

Other than the long running iterators, as pointed by @ibrahim, the vlogMaxEntries was set with default value of 10000 in the version of badger on which the test was run. This resulted each vlog file to contain only data of approximately 15 MB but each vlog file is mmaped to 2GB. After setting vlogMaxEntries to 1000000 we didn’t encounter OOM.

github.com

dgraph-io/badger/blob/763e7d7303f616fb4cfd36e6e21110ec5649df39/value.go#L1021


	if err = lf.bootstrap(); err != nil {
		removeFile()
		return nil, err
	}

	if err = syncDir(vlog.dirPath); err != nil {
		removeFile()
		return nil, errFile(err, vlog.dirPath, "Sync value log dir")
	}

	if err = lf.mmap(2 * vlog.opt.ValueLogFileSize); err != nil {
		removeFile()
		return nil, errFile(err, lf.path, "Mmap value log file")
	}

	vlog.filesLock.Lock()
	vlog.filesMap[fid] = lf
	vlog.maxFid = fid
	// writableLogOffset is only written by write func, by read by Read func.
	// To avoid a race condition, all reads and updates to this variable must be
	// done via atomics.

Topic		Replies	Views
Badger performance and memory usage recommendations Badger	9	5390	June 27, 2020
Provide simple option for limiting total memory usage Issues badger , kind:enhancement , status:accepted , area:documentation	19	2085	May 1, 2022
Crash with memtable sizes over 64 << 24 Badger	1	624	November 7, 2021
Badger compaction results in OOM and loss of data Badger	7	1580	November 21, 2020
Badger write performance on 1 TB data Badger	4	2806	March 15, 2022

Memtable memory usage causes OOM

Related topics