[WIP] Performance bottleneck on 1 TB data

Dev
9

Update: 14/07/2020

I reran Dgraph after previous crash.

Throughput numbers:

Time elapsed: 66894, Taskname: insert-person, Speed: 117
Time elapsed: 66894, Taskname: insert-friend, Speed: 114
Time elapsed: 66894, Taskname: get-friends-two-hop, Speed: 94
Time elapsed: 66895, Taskname: insert-person, Speed: 115
Time elapsed: 66895, Taskname: get-friends-one-hop, Speed: 100
Time elapsed: 66895, Taskname: insert-friend, Speed: 112
Time elapsed: 66895, Taskname: get-friends-two-hop, Speed: 90
Time elapsed: 66896, Taskname: get-friends-one-hop, Speed: 102
Time elapsed: 66896, Taskname: insert-friend, Speed: 111
Time elapsed: 66896, Taskname: insert-person, Speed: 117
Time elapsed: 66896, Taskname: get-friends-two-hop, Speed: 94
Time elapsed: 66897, Taskname: get-friends-one-hop, Speed: 96
Time elapsed: 66897, Taskname: insert-person, Speed: 114
Time elapsed: 66897, Taskname: insert-friend, Speed: 104
Time elapsed: 66897, Taskname: get-friends-two-hop, Speed: 96
Time elapsed: 66898, Taskname: get-friends-one-hop, Speed: 95
Time elapsed: 66898, Taskname: insert-person, Speed: 124
Time elapsed: 66898, Taskname: insert-friend, Speed: 123

Directory sizes

54G	./p
4.1M	./w
2.4G	./zw

RES usage shown in htop

16GB

Profile files for Heap and CPU

mem.pprof (259.9 KB) cpu.pprof (110.7 KB)
heap top:

File: dgraph
Build ID: 371190db1475fb14a7dfd3c44905e712b120ca09
Type: inuse_space
Time: Jul 14, 2020 at 9:39am (UTC)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top20
Showing nodes accounting for 1016.81MB, 97.43% of 1043.60MB total
Dropped 144 nodes (cum <= 5.22MB)
Showing top 20 nodes out of 84
      flat  flat%   sum%        cum   cum%
  384.79MB 36.87% 36.87%   384.79MB 36.87%  github.com/dgraph-io/ristretto.newCmRow (inline)
  292.10MB 27.99% 64.86%   292.10MB 27.99%  github.com/dgraph-io/ristretto/z.(*Bloom).Size (inline)
  166.41MB 15.95% 80.81%   166.41MB 15.95%  github.com/dgraph-io/badger/v2/skl.newArena (inline)
      69MB  6.61% 87.42%       69MB  6.61%  github.com/dgraph-io/badger/v2/table.NewTableBuilder
   44.63MB  4.28% 91.70%    51.37MB  4.92%  github.com/dgraph-io/dgraph/worker.(*node).processApplyCh.func1
   41.22MB  3.95% 95.65%    51.72MB  4.96%  github.com/dgraph-io/badger/v2/pb.(*TableIndex).Unmarshal
   10.50MB  1.01% 96.65%    10.50MB  1.01%  github.com/dgraph-io/badger/v2/pb.(*BlockOffset).Unmarshal
    6.89MB  0.66% 97.31%     6.89MB  0.66%  github.com/dgraph-io/badger/v2/table.OpenTable
    1.27MB  0.12% 97.43%   580.10MB 55.59%  github.com/dgraph-io/ristretto.NewCache
         0     0% 97.43%    49.40MB  4.73%  github.com/dgraph-io/badger/v2.(*DB).RunValueLogGC
         0     0% 97.43%   166.41MB 15.95%  github.com/dgraph-io/badger/v2.(*DB).doWrites.func1
         0     0% 97.43%   166.41MB 15.95%  github.com/dgraph-io/badger/v2.(*DB).ensureRoomForWrite
         0     0% 97.43%    48.90MB  4.69%  github.com/dgraph-io/badger/v2.(*DB).get
         0     0% 97.43%   166.41MB 15.95%  github.com/dgraph-io/badger/v2.(*DB).writeRequests
         0     0% 97.43%    49.24MB  4.72%  github.com/dgraph-io/badger/v2.(*Iterator).Seek
         0     0% 97.43%    46.86MB  4.49%  github.com/dgraph-io/badger/v2.(*IteratorOptions).pickTables
         0     0% 97.43%    51.65MB  4.95%  github.com/dgraph-io/badger/v2.(*Txn).NewIterator
         0     0% 97.43%    51.65MB  4.95%  github.com/dgraph-io/badger/v2.(*Txn).NewKeyIterator
         0     0% 97.43%    51.65MB  4.95%  github.com/dgraph-io/badger/v2.(*levelHandler).appendIterators
         0     0% 97.43%    48.90MB  4.69%  github.com/dgraph-io/badger/v2.(*levelHandler).get