Bulk loader xidmap memory optimization

ibrahim · March 3, 2021, 1:21pm

We didn’t implement this. We switched over to a B+ tree

dgraph-io/dgraph/blob/c4c3248db7a2a914171850430953c1890e3d7b75/xidmap/xidmap.go#L58


	wg     sync.WaitGroup

	kvBuf  []kv
	kvChan chan []kv
}

type shard struct {
	sync.RWMutex
	block

	tree *z.Tree
}

type block struct {
	start, end uint64
}

type kv struct {
	key, value []byte
}

This B+ tree was earlier mmaped to conserve memory but now we keep it completely in memory

Topic		Replies	Views
Bulk Loader transaction scope (blank node identification) Dgraph kind:question , dgraph , kind:bug	6	639	February 1, 2021
Bulk loader OOM Users	4	900	August 10, 2020
Bulk loader -x option Users mutation	7	817	May 9, 2020
Improve throughput of bulk loader with distributed loading Dgraph dgraph , kind:enhancement , priority:p2 , status:accepted , popular	21	1032	February 6, 2020
Where is the mapping of xids to uids which is created by bulk Users	3	660	April 5, 2018

Bulk loader xidmap memory optimization

Related topics