Database open time is really high

mrjn · January 3, 2019, 12:27am

The number of files definitely makes a difference. Badger picks up an index from each SST file, which looks like it is being done serially, so it takes time.

github.com

dgraph-io/badger/blob/a7db8195d90af472130f03e46a00ba0b5b9c2cf0/levels.go#L106


	}

	// Compare manifest against directory, check for existent/non-existent files, and remove.
	if err := revertToManifest(kv, mf, getIDMap(kv.opt.Dir)); err != nil {
		return nil, err
	}

	// Some files may be deleted. Let's reload.
	tables := make([][]*table.Table, kv.opt.MaxLevels)
	var maxFileID uint64
	for fileID, tableManifest := range mf.Tables {
		fname := table.NewFilename(fileID, kv.opt.Dir)
		var flags uint32 = y.Sync
		if kv.opt.ReadOnly {
			flags |= y.ReadOnly
		}
		fd, err := y.OpenExistingFile(fname, flags)
		if err != nil {
			closeAllTables(tables)
			return nil, errors.Wrapf(err, "Opening file: %q", fname)
		}

Maybe we could look into reading these tables concurrently, and see if that makes a difference.

Could you share the output of badger info here, so I can try and replicate your setup? Also, a list of your options, unless you’re using DefaultOptions.

Topic		Replies	Views
Figuring optimal options for usage pattern App Development badger	2	635	January 12, 2021
Jaeger can not start, Badger deleting file very slow, jaeger can not init storage Badger kind:question	7	1623	November 30, 2021
Badger read performance on 1TB data Badger	0	716	August 27, 2020
Badger write performance on 1 TB data Badger	4	2806	March 15, 2022
Badger getting slower as table gets bigger Badger	0	532	November 18, 2021

Database open time is really high

Related topics