LOG Compact FAILED with error: MANIFEST removes non-existing table 15777621,

We encountered a very similar situation, the 3 member cluster became non responsive, while the leader was trying to compact and mostly working on (bugged down by) the same two files. Here are the warning messages, showing up every second (or very two seconds):

While running doCompact: MANIFEST removes non-existing table 8603827
...
While running doCompact: MANIFEST removes non-existing table 8603684

I can see those two files under the /var/lib/dgraph/p/ , and sitting there for 5 hours so far…

/var/lib/dgraph/p/8603827.sst (9:11 - Sep 13 09:11)
/var/lib/dgraph/p/8603684.sst (9:11 - Sep 13 09:11)

The only relevant errors before 9:11 are:

I0913 08:55:29.157530    7078 draft.go:1530] Found 1 old transactions. Acting to abort them.
I0913 08:55:29.159236    7078 draft.go:1491] TryAbort 1 txns with start ts. Error: <nil>
I0913 08:55:29.159249    7078 draft.go:1514] TryAbort selectively proposing only aborted txns: txns:<start_ts:24886942 >
I0913 08:55:29.160094    7078 draft.go:1533] Done abortOldTransactions for 1 txns. Error: <nil>

And at 9:11, kaboom!

 W0913 09:11:09.175527    7078 log.go:36] [Compactor: 2] LOG Compact FAILED with error: sync /var/lib/dgraph/p/MANIFEST: file already closed:
W0913 09:11:09.181562    7078 log.go:36] While running doCompact: sync /var/lib/dgraph/p/MANIFEST: file already closed
 W0913 09:11:09.228910    7078 log.go:36] [Compactor: 1] LOG Compact FAILED with error: MANIFEST removes non-existing table 8603684
W0913 09:11:09.229006    7078 log.go:36] While running doCompact: MANIFEST removes non-existing table 8603684

I would love to remove those two files, and see if the series of complaints continue! :heart:
Or should we mark the leader as sick, remove it from the cluster, allow the other two members to elect a leader, clear up the data on the faulty sick member, and add it back into the cluster? We are still at consultation level. How did you guys fix this problem in the past? Thanks, Steve

2 Likes