While bulk loading a dataset with a high number of unique xids, the bulk loader goes out of memory in the mapper phase. We keep a sharded map (XidMap) of xids to their corresponding id so that multiple instances of the same blank nodes get the same id. The collected memory required by the shared map…

Bulk loader xidmap memory optimization

Paras (Paras Shah) July 3, 2020, 4:29pm 5

For (3), yes. But for (2), code can figure out the needed values.

Interesting. Need to read more about this.

Topic		Replies	Views
Bulk Loader transaction scope (blank node identification) Dgraph kind:question , dgraph , kind:bug	6	639	February 1, 2021
Bulk loader OOM Users	4	900	August 10, 2020
Bulk loader -x option Users mutation	7	817	May 9, 2020
Improve throughput of bulk loader with distributed loading Dgraph dgraph , kind:enhancement , priority:p2 , status:accepted , popular	21	1032	February 6, 2020
Where is the mapping of xids to uids which is created by bulk Users	3	660	April 5, 2018