Critical bug in v21.12 permanently crashloops whole groups

miyamoto.kaoru · May 19, 2023, 4:57pm

This critical issue seems to be still open.
Is it because it’s a dicontinued 21.12.0 specific problem?

Panic observed during applyCh

opened 01:08AM - 05 Jan 22 UTC

kind/bug investigate area/crash community

Github issues are back! Putting in this one now: Following on topic here: ht…tps://discuss.dgraph.io/t/critical-bug-in-v21-12-permanently-crashloops-whole-groups/16383/7 A panic is continually observed here: ``` 2021/12/22 06:56:16 Unable to find txn with start ts: 2178082 github.com/dgraph-io/dgraph/x.AssertTruef /ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:107 github.com/dgraph-io/dgraph/worker.(*node).applyMutations /ext-go/1/src/github.com/dgraph-io/dgraph/worker/draft.go:707 github.com/dgraph-io/dgraph/worker.(*node).applyCommitted /ext-go/1/src/github.com/dgraph-io/dgraph/worker/draft.go:744 github.com/dgraph-io/dgraph/worker.(*node).processApplyCh.func1 /ext-go/1/src/github.com/dgraph-io/dgraph/worker/draft.go:931 github.com/dgraph-io/dgraph/worker.(*node).processApplyCh.func2 /ext-go/1/src/github.com/dgraph-io/dgraph/worker/draft.go:970 github.com/dgraph-io/dgraph/worker.(*node).processApplyCh /ext-go/1/src/github.com/dgraph-io/dgraph/worker/draft.go:1025 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1581 ``` If this panic occurs, the node will restart and replay its WAL - and will encounter the same issue again, permanently disabling the node. This tends to happen to entire groups at once. v21.12: ``` $ dgraph version Dgraph version : v21.12.0 Dgraph codename : zion Dgraph SHA-256 : 078c75df9fa1057447c8c8afc10ea57cb0a29dfb22f9e61d8c334882b4b4eb37 Commit SHA-1 : d62ed5f15 Commit timestamp : 2021-12-02 21:20:09 +0530 Branch : HEAD Go version : go1.17.3 jemalloc enabled : true ```

Can I avoid this problem if I upgrade to version 22.0 .x or higher?

Topic		Replies	Views
What's the point is to put this message every few milliseconds? Dgraph	19	1122	February 6, 2019
Unable to reach leader in group 1 - dir structures help Dgraph	38	1870	April 11, 2023
Dgraph can't idle without being oomkilled after large data ingestion Dgraph	63	3849	September 14, 2020
Help needed: dgraph keeps crashing under load - what s going on? Dgraph	4	1843	April 13, 2020
LOG Compact FAILED with error: MANIFEST removes non-existing table 15777621, Dgraph Cloud dgraph	19	1678	October 2, 2021

Critical bug in v21.12 permanently crashloops whole groups

Related topics