chen
February 16, 2018, 11:33am
1
Hello,
String schema = " phone_number: string @index(exact) .\n"
+ " taobao_name: string @index(term) .\n"
+ " contact_name: string @index(term) .\n"
+ " contact: uid @reverse .\n"
+ " called: uid @reverse .\n"
+ " is_black: int @index(int) .\n";
The performance of bulk load seems very bad.
chen
February 16, 2018, 11:52am
2
For the second try, I encountered exception
2018/02/16 19:50:57 batch.go:133: Error while mutating rpc error: code = Unknown desc = Uid: [721136] cannot be greater than lease: [10000]
2018/02/16 19:50:57 batch.go:133: Error while mutating rpc error: code = Unknown desc = Uid: [320208] cannot be greater than lease: [10000]
2018/02/16 19:50:57 batch.go:133: Error while mutating rpc error: code = Unknown desc = Uid: [154871] cannot be greater than lease: [10000]
2018/02/16 19:50:57 batch.go:133: Error while mutating rpc error: code = Unknown desc = Uid: [10001] cannot be greater than lease: [10000]
2018/02/16 19:50:57 batch.go:133: Error while mutating rpc error: code = Unknown desc = Uid: [622675] cannot be greater than lease: [10000]
2018/02/16 19:50:57 batch.go:133: Error while mutating rpc error: code = Unknown desc = Uid: [912974] cannot be greater than lease: [10000]
2018/02/16 19:50:58 batch.go:133: Error while mutating rpc error: code = Unknown desc = Uid: [1154532] cannot be greater than lease: [10000]
2018/02/16 19:50:58 batch.go:133: Error while mutating rpc error: code = Unknown desc = Uid: [1444275] cannot be greater than lease: [10000]
2018/02/16 19:50:58 batch.go:133: Error while mutating rpc error: code = Unknown desc = Uid: [1064224] cannot be greater than lease: [10000]
2018/02/16 19:50:58 batch.go:133: Error while mutating rpc error: code = Unknown desc = Uid: [950474] cannot be greater than lease: [10000]
2018/02/16 19:50:58 batch.go:133: Error while mutating rpc error: code = Unknown desc = Uid: [862013] cannot be greater than lease: [10000]
2018/02/16 19:50:58 batch.go:133: Error while mutating rpc error: code = Unknown desc = Uid: [1203869] cannot be greater than lease: [10000]
2018/02/16 19:50:58 batch.go:133: Error while mutating rpc error: code = Unknown desc = Uid: [891304] cannot be greater than lease: [10000]
2018/02/16 19:50:58 batch.go:133: Error while mutating rpc error: code = Unknown desc = Uid: [491179] cannot be greater than lease: [10000]
2018/02/16 19:50:58 batch.go:133: Error while mutating rpc error: code = Unknown desc = Uid: [801645] cannot be greater than lease: [10000]
2018/02/16 19:50:58 batch.go:133: Error while mutating rpc error: code = Unknown desc = Uid: [1243842] cannot be greater than lease: [10000]
2018/02/16 19:50:58 batch.go:133: Error while mutating rpc error: code = Unknown desc = Uid: [430933] cannot be greater than lease: [10000]
2018/02/16 19:50:58 batch.go:133: Error while mutating rpc error: code = Unknown desc = Uid: [1613466] cannot be greater than lease: [10000]
2018/02/16 19:50:58 batch.go:133: Error while mutating rpc error: code = Unknown desc = Uid: [1462474] cannot be greater than lease: [10000]
2018/02/16 19:50:58 batch.go:133: Error while mutating rpc error: code = Unknown desc = Uid: [101525] cannot be greater than lease: [10000]
2018/02/16 19:50:58 batch.go:133: Error while mutating rpc error: code = Unknown desc = Uid: [1822944] cannot be greater than lease: [10000]
2018/02/16 19:50:58 batch.go:133: Error while mutating rpc error: code = Unknown desc = Uid: [1892714] cannot be greater than lease: [10000]
2018/02/16 19:50:58 batch.go:133: Error while mutating rpc error: code = Unknown desc = Uid: [1781223] cannot be greater than lease: [10000]
2018/02/16 19:50:58 batch.go:133: Error while mutating rpc error: code = Unknown desc = Uid: [1323309] cannot be greater than lease: [10000]
I have tried delete the whole database and retry bulk load, still meet such issue.
For the first question? what is the value you set for --map_shards or provide us the command you used for bulk loading?
For the second one, you have to delete tmp folder in the working directory of dgraph bulk.
chen
February 23, 2018, 7:40am
4
dgraph live --rdfs *.rdf.gz --zero localhost:5080
chen
February 23, 2018, 8:01am
5
bulk load now works.
madings-MacBook-Pro:data mading$ dgraph bulk -r dgraph_sample.rdf.gz -s draph.schema --map_shards=4 --reduce_shards=1 --http localhost:8000 --zero_addr=localhost:5080
{
"RDFDir": "dgraph_sample.rdf.gz",
"SchemaFile": "draph.schema",
"DgraphsDir": "out",
"TmpDir": "tmp",
"NumGoroutines": 4,
"MapBufSize": 67108864,
"ExpandEdges": true,
"SkipMapPhase": false,
"CleanupTmp": true,
"NumShufflers": 1,
"Version": false,
"StoreXids": false,
"ZeroAddr": "localhost:5080",
"HttpAddr": "localhost:8000",
"MapShards": 4,
"ReduceShards": 1
}
The bulk loader needs to open many files at once. This number depends on the size of the data set loaded, the map file output size, and the level of indexing. 100,000 is adequate for most data set sizes. See `man ulimit` for details of how to change the limit.
Current max open files limit: 7168
2018/02/23 15:58:58 loader.go:87: Connecting to zero at localhost:5080
MAP 01s rdf_count:87.22k rdf_speed:86.71k/sec edge_count:585.0k edge_speed:581.6k/sec
MAP 02s rdf_count:263.4k rdf_speed:131.0k/sec edge_count:1.821M edge_speed:905.5k/sec
MAP 03s rdf_count:428.4k rdf_speed:142.0k/sec edge_count:3.001M edge_speed:995.0k/sec
MAP 04s rdf_count:528.1k rdf_speed:131.5k/sec edge_count:3.711M edge_speed:923.9k/sec
MAP 05s rdf_count:667.7k rdf_speed:132.9k/sec edge_count:4.692M edge_speed:934.3k/sec
MAP 06s rdf_count:766.6k rdf_speed:127.2k/sec edge_count:5.338M edge_speed:885.8k/sec
MAP 07s rdf_count:868.9k rdf_speed:123.6k/sec edge_count:5.713M edge_speed:812.7k/sec
MAP 08s rdf_count:951.8k rdf_speed:118.5k/sec edge_count:5.962M edge_speed:742.1k/sec
MAP 09s rdf_count:1.045M rdf_speed:115.6k/sec edge_count:6.240M edge_speed:690.5k/sec
MAP 10s rdf_count:1.155M rdf_speed:115.1k/sec edge_count:6.571M edge_speed:654.7k/sec
chen
February 23, 2018, 8:19am
6
still encountered exception : Unknown desc = Uid: [1243842] cannot be greater than lease: [10000] if live load is used
Live loader uses ‘x’ folder to store UID mappings, if you are fresh loading you probably have to delete this folder.
1 Like
system
March 25, 2018, 2:52pm
8
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.