Hey, those logs are not from Bulkload. They are from LiveLoad. If you use bulkload you will have more speed on data insertion. Check Loading close to 1M edges/sec into Dgraph - Dgraph Blog
If you do not want to increase memory, I recommend using Bulkload for data insertion.
And which of the problems presented still persist?
A bulkload log looks like this:
./dgraph bulk -r out.rdf.gz -s movielens.schema --map_shards=4 --reduce_shards=1 --zero localhost:5080
{
"RDFDir": "out.rdf.gz",
"SchemaFile": "movielens.schema",
"DgraphsDir": "out",
"TmpDir": "tmp",
"NumGoroutines": 8,
"MapBufSize": 67108864,
"ExpandEdges": true,
"SkipMapPhase": false,
"CleanupTmp": true,
"NumShufflers": 1,
"Version": false,
"StoreXids": false,
"ZeroAddr": "localhost:5080",
"HttpAddr": "localhost:8080",
"MapShards": 4,
"ReduceShards": 1
}
The bulk loader needs to open many files at once. This number depends on the size of the data
set loaded, the map file output size, and the level of indexing. 100,000 is adequate for most data
set sizes. See `man ulimit` for details of how to change the limit.
Current max open files limit: 7168
2018/07/11 21:55:13 loader.go:77: Connecting to zero at localhost:5080
MAP 01s rdf_count:30.54k rdf_speed:29.83k/sec edge_count:94.38k edge_speed:92.17k/sec
MAP 02s rdf_count:108.4k rdf_speed:53.44k/sec edge_count:327.9k edge_speed:161.7k/sec
2018/07/11 21:55:15 merge_shards.go:36: Shard tmp/shards/002 -> Reduce tmp/shards/shard_0/002
2018/07/11 21:55:15 merge_shards.go:36: Shard tmp/shards/001 -> Reduce tmp/shards/shard_0/001
2018/07/11 21:55:15 merge_shards.go:36: Shard tmp/shards/000 -> Reduce tmp/shards/shard_0/000
2018/07/11 21:55:15 merge_shards.go:36: Shard tmp/shards/003 -> Reduce tmp/shards/shard_0/003
REDUCE 03s [100.00%] edge_count:327.9k edge_speed:327.9k/sec plist_count:16.63k plist_speed:16.63k/sec
REDUCE 03s [100.00%] edge_count:327.9k edge_speed:877.8k/sec plist_count:16.63k plist_speed:44.52k/sec
Total: 03s
wich way? like this way? apollo-universal-starter-kit-With-Dgraph-DB/packages/server/src/dgraph/dgraphconnector.js at 86160bae53595290703221192f1b7a405ae8bc38 · OpenDgraph/apollo-universal-starter-kit-With-Dgraph-DB · GitHub