High IO on dgraph nodes

Users
1

I am trying to figure out reason for increased data transfer between dgraph nodes. Each node is transferring over 500 Kbps to other two nodes. In metrics exposed by dgraph I can see that our number of mutation has increased to 700 ops (all), which I hope isn’t a lot.
Logs keep spewing following (on node 2), which is causing us io problems:

2017/12/08 07:25:54 node.go:554: RECEIVED: MsgApp 3-->2
2017/12/08 07:25:54 node.go:554: RECEIVED: MsgApp 3-->2
2017/12/08 07:25:54 node.go:554: RECEIVED: MsgApp 3-->2
2017/12/08 07:25:54 node.go:554: RECEIVED: MsgApp 3-->2
2017/12/08 07:25:54 node.go:554: RECEIVED: MsgApp 3-->2
2017/12/08 07:25:54 node.go:177: SENDING: MsgAppResp 2-->3
2017/12/08 07:25:54 node.go:177: SENDING: MsgAppResp 2-->3
2017/12/08 07:25:54 node.go:177: SENDING: MsgAppResp 2-->3
2017/12/08 07:25:54 node.go:177: SENDING: MsgAppResp 2-->3
2017/12/08 07:25:54 node.go:177: SENDING: MsgAppResp 2-->3
2017/12/08 07:25:54 node.go:177: SENDING: MsgProp 2-->3

in iotop I see, which is likely because of lots of log writes:

IO> Command
66% [jbd2/xvda1-8]

Any kind of help would be really appreciated.

2

The data is consistently replicated on all nodes so the data transfered between nodes would depend upon the number of mutations you are doing. Does your mutations have long string values ?
Can you try disabling logs and check the IO usage and also can you please file an issue on github regarding logging. We could work on improving it.

The high io might not be due to logs. You can get an estimate of the io done by dgraph by checking the size of p and w directly every 10 seconds.

3

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.