[WIP] Performance bottleneck on 1 TB data

Dev
24

Experiments with Indexing

While inserting data into Dgraph, I didn’t have any schema. So everything got inserted with default types. After insertions, my schema was below:

<AboutMe>: default .
<Author>: [uid] .
<Body>: [uid] .
<Chosen.Answer>: [uid] .
<Comment>: [uid] .
<CreationDate>: default .
<DisplayName>: default .
<Downvote>: [uid] .
<Has.Answer>: [uid] .
<Id>: default .
<LastAccessDate>: default .
<Location>: default .
<Owner>: [uid] .
<Post>: [uid] .
<PostCount>: default .
<Reputation>: default .
<Score>: default .
<Tag.Text>: default .
<Tag>: [uid] .
<TagName>: default .
<Text>: default .
<Timestamp>: default .
<Title>: [uid] .
<Type>: default .
<Upvote>: [uid] .
<ViewCount>: default .
<asfdasdfljals>: string .
<dgraph.graphql.schema>: string .
<dgraph.graphql.xid>: string @index(exact) @upsert .

I performed below indexing changes, with their time of completion:

| Schema operation                   | Time  | Completed |
|------------------------------------|-------|-----------|
| <Tag.Text>: string @index(hash)    | 1m    | Yes       |
| <Author>: [uid] @reverse .         | 1h15m | Yes       |
| <Tag>: [uid] @reverse              | 15m   | Yes       |
| <Downvote>: [uid] @reverse         | 3m    | Yes       |
| <Upvote>: [uid] @reverse           | 20m   | Yes       |
| <Text>: string @index(fulltext)    | 2h30m | No        |

While performing indexing for <Text>: string @index(fulltext), Dgraph got crashed. It got crashed and restart resulted in below error:

I0806 11:30:12.819371   18401 log.go:34] 6851 tables out of 6954 opened in 6m15.057s
I0806 11:30:15.811030   18401 log.go:34] 6903 tables out of 6954 opened in 6m18.048s
I0806 11:30:18.500677   18401 log.go:34] All 6954 tables opened in 6m20.738s
[Sentry] 2020/08/06 11:30:19 Sending fatal event [b3b9b8dac9c94c92b0f727d9c948fd29] to o318308.ingest.sentry.io project: 5208688
2020/08/06 11:30:19 Inter: Biggest(j-1) 
00000000  00 00 0e 4c 61 73 74 41  63 63 65 73 73 44 61 74  |...LastAccessDat|
00000010  65 00 00 00 00 00 16 a1  72 cc ff ff ff ff ff be  |e.......r.......|
00000020  17 84                                             |..|

 vs Smallest(j): 
00000000  00 00 04 42 6f 64 79 00  00 00 00 00 08 37 dc cc  |...Body......7..|
00000010  ff ff ff ff ff e1 e6 d6                           |........|

: level=3 j=5 numTables=1351
github.com/dgraph-io/badger/v2.(*levelHandler).validate
	/home/ashish/projects/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200615081930-c45d966681d4/util.go:55
github.com/dgraph-io/badger/v2.(*levelsController).validate
	/home/ashish/projects/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200615081930-c45d966681d4/util.go:33
github.com/dgraph-io/badger/v2.newLevelsController
	/home/ashish/projects/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200615081930-c45d966681d4/levels.go:191
github.com/dgraph-io/badger/v2.Open
	/home/ashish/projects/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200615081930-c45d966681d4/db.go:354
github.com/dgraph-io/badger/v2.OpenManaged
	/home/ashish/projects/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200615081930-c45d966681d4/managed_db.go:26
github.com/dgraph-io/dgraph/worker.(*ServerState).initStorage
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/worker/server_state.go:171
github.com/dgraph-io/dgraph/worker.InitServerState
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/worker/server_state.go:54
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:655
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.init.2.func1
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:95
github.com/spf13/cobra.(*Command).execute
	/home/ashish/projects/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
	/home/ashish/projects/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
	/home/ashish/projects/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:70
main.main
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/main.go:78
runtime.main
	/home/ashish/go/src/runtime/proc.go:203
runtime.goexit
	/home/ashish/go/src/runtime/asm_amd64.s:1373
Levels Controller
github.com/dgraph-io/badger/v2.(*levelsController).validate
	/home/ashish/projects/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200615081930-c45d966681d4/util.go:34
github.com/dgraph-io/badger/v2.newLevelsController
	/home/ashish/projects/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200615081930-c45d966681d4/levels.go:191
github.com/dgraph-io/badger/v2.Open
	/home/ashish/projects/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200615081930-c45d966681d4/db.go:354
github.com/dgraph-io/badger/v2.OpenManaged
	/home/ashish/projects/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200615081930-c45d966681d4/managed_db.go:26
github.com/dgraph-io/dgraph/worker.(*ServerState).initStorage
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/worker/server_state.go:171
github.com/dgraph-io/dgraph/worker.InitServerState
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/worker/server_state.go:54
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:655
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.init.2.func1
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:95
github.com/spf13/cobra.(*Command).execute
	/home/ashish/projects/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
	/home/ashish/projects/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
	/home/ashish/projects/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:70
main.main
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/main.go:78
runtime.main
	/home/ashish/go/src/runtime/proc.go:203
runtime.goexit
	/home/ashish/go/src/runtime/asm_amd64.s:1373
Level validation
github.com/dgraph-io/badger/v2.newLevelsController
	/home/ashish/projects/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200615081930-c45d966681d4/levels.go:193
github.com/dgraph-io/badger/v2.Open
	/home/ashish/projects/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200615081930-c45d966681d4/db.go:354
github.com/dgraph-io/badger/v2.OpenManaged
	/home/ashish/projects/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200615081930-c45d966681d4/managed_db.go:26
github.com/dgraph-io/dgraph/worker.(*ServerState).initStorage
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/worker/server_state.go:171
github.com/dgraph-io/dgraph/worker.InitServerState
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/worker/server_state.go:54
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:655
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.init.2.func1
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:95
github.com/spf13/cobra.(*Command).execute
	/home/ashish/projects/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
	/home/ashish/projects/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
	/home/ashish/projects/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:70
main.main
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/main.go:78
runtime.main
	/home/ashish/go/src/runtime/proc.go:203
runtime.goexit
	/home/ashish/go/src/runtime/asm_amd64.s:1373
Error while creating badger KV posting store
github.com/dgraph-io/dgraph/x.Checkf
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/x/error.go:51
github.com/dgraph-io/dgraph/worker.(*ServerState).initStorage
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/worker/server_state.go:172
github.com/dgraph-io/dgraph/worker.InitServerState
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/worker/server_state.go:54
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:655
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.init.2.func1
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:95
github.com/spf13/cobra.(*Command).execute
	/home/ashish/projects/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
	/home/ashish/projects/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
	/home/ashish/projects/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:70
main.main
	/home/ashish/projects/src/github.com/dgraph-io/dgraph/dgraph/main.go:78
runtime.main
	/home/ashish/go/src/runtime/proc.go:203
runtime.goexit
	/home/ashish/go/src/runtime/asm_amd64.s:1373
1 Like