Help needed: dgraph keeps crashing under load - what s going on?

[mrjn (Manish R Jain)]

@stroboshaver

rpc error: code = Unknown desc = Assigning IDs is only allowed on leader.

This is a temporary error. It just indicates that Zero leadership changed and Alpha needs to connect to a new Zero leader.

@stroboshaver

code = Unknown desc = Pending transactions found. Please retry operation

Again, a temporary error as it says.

@stroboshaver

I0406 12:26:54.120364 280 groups.go:872] Zero leadership changed. Renewing oracle delta stream.
I0406 12:27:02.794360 280 groups.go:808] Leader idx=0x1 of group=1 is connecting to Zero for txn updates
I0406 12:27:02.803518 280 groups.go:817] Got Zero leader: localhost:5080
E0406 12:27:02.795405 280 groups.go:848] Error in oracle delta stream. Error: rpc error: code = Canceled desc = context canceled

This one shows that Zero leadership changed. So, Alpha canceled the Zero stream which spits out this error log. It will then reconnect to the new Zero leader.

@stroboshaver

Sometimes Dgraph was in a very annoying state after crash, where it was in an endless loop of trying to playback the files. i had to rm -rf all the time.

Dgraph would replay the Raft WAL logs on a crash, that’s normal. Deleting them with a rm -rf would cause issues – unless you’re just starting a new cluster.

I’m not seeing any logs here which resembles a crash – it sounds like you might be going OOM or something. Also, any reason to not use 20.03, which is the latest release? This looks like an older version.

CC: @Paras @martinmr