I’m using bulk loader to load a 25GB JSON file with elements like the following, and a blank schema file.
[
{
"uid": "_:b9e10e568ae9a1edb1f82dc95584d830fd096309",
"title": "Sitjumaq na sapitj",
"dgraph.type": "Article",
"links": [
{
"uid": "_:26018315a880b1dcffd41c0556e72bd12dfc924c",
"title": "sitjumaq na sapitj",
"dgraph.type": "Article",
"links|weight": 5
}
]
},
{
"uid": "_:b4c9fc147c13ca9158ca607bb44f87ba125575ce",
"title": "Akira Suzuki",
"dgraph.type": "Article",
"links": [
{
"uid": "_:a38d83577c4c934221810e5b8b706df1f649ce4d",
"title": "Dripun",
"dgraph.type": "Article",
"links|weight": 5
}
]
}
]
I get this error at about 287.3 M edges generated every time i run the loader:
While processing 'val(Unix)|{{codett', brackets are not closed properly
github.com/dgraph-io/dgraph/chunker.handleBasicType
/ext-go/1/src/github.com/dgraph-io/dgraph/chunker/json_parser.go:234
github.com/dgraph-io/dgraph/chunker.(*NQuadBuffer).mapToNquads
/ext-go/1/src/github.com/dgraph-io/dgraph/chunker/json_parser.go:536
github.com/dgraph-io/dgraph/chunker.(*NQuadBuffer).mapToNquads
/ext-go/1/src/github.com/dgraph-io/dgraph/chunker/json_parser.go:637
github.com/dgraph-io/dgraph/chunker.(*NQuadBuffer).ParseJSON
/ext-go/1/src/github.com/dgraph-io/dgraph/chunker/json_parser.go:775
github.com/dgraph-io/dgraph/chunker.(*jsonChunker).Parse
/ext-go/1/src/github.com/dgraph-io/dgraph/chunker/chunk.go:308
github.com/dgraph-io/dgraph/dgraph/cmd/bulk.(*mapper).run.func1
/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/mapper.go:226
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1581
github.com/dgraph-io/dgraph/x.Check
/ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:42
github.com/dgraph-io/dgraph/dgraph/cmd/bulk.(*mapper).run.func1
/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/mapper.go:229
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1581
Trying with smaller files with the exact same structure gives no errors whatsoever and load completes successfully.
Since i’m not able to open the actual file due to its size i have no clue about what could be causing this error.
Dgraph metadata
dgraph version
Dgraph version : v21.12.0
Dgraph codename : zion
Dgraph SHA-256 : 078c75df9fa1057447c8c8afc10ea57cb0a29dfb22f9e61d8c334882b4b4eb37
Commit SHA-1 : d62ed5f15
Commit timestamp : 2021-12-02 21:20:09 +0530
Branch : HEAD
Go version : go1.17.3
jemalloc enabled : true