Did I lose my data for forever?

Issues Dgraph
,
20

UPD:
I disabled google cloud persistent disk and run it just on node disks.

Nothing changed.
In 3 days I got the same issues again:

W0614 13:59:18.326329 20 log.go:36] [Compactor: 2] LOG Compact FAILED with error: MANIFEST removes non-existing table 113984: {span:0xc0097be280 compactorId:2 t:{baseLevel:3 targetSz:[0 10485760 10485760 10485760 18157929 181579297 1815792975] fileSz:[67108864 2097152 2097152 2097152 4194304 8388608 16777216]} p:{level:3 score:23.36615171432495 adjusted:21.054207908706115 dropPrefixes:[] t:{baseLevel:3 targetSz:[0 10485760 10485760 10485760 18157929 181579297 1815792975] fileSz:[67108864 2097152 2097152 2097152 4194304 8388608 16777216]}} thisLevel:0xc0000d3140 nextLevel:0xc0000d31a0 top:[0xc016b760c0] bot:[0xc01302fd40 0xc00d15d5c0] thisRange:{left:[0 0 0 0 0 0 0 0 0 0 17 101 98 97 121 46 108 105 115 116 105 110 103 46 110 97 109 101 2 10 97 110 102 0 0 0 0 0 0 0 0] right:[0 0 0 0 0 0 0 0 0 0 17 101 98 97 121 46 108 105 115 116 105 110 103 46 110 97 109 101 2 10 97 114 116 255 255 255 255 255 255 255 255] inf:false size:0} nextRange:{left:[0 0 0 0 0 0 0 0 0 0 17 101 98 97 121 46 108 105 115 116 105 110 103 46 110 97 109 101 2 10 97 103 117 0 0 0 0 0 0 0 0] right:[0 0 0 0 0 0 0 0 0 0 17 101 98 97 121 46 108 105 115 116 105 110 103 46 110 97 109 101 2 10 99 101 108 255 255 255 255 255 255 255 255] inf:false size:0} splits:[] thisSize:2124381 dropPrefixes:[]}
W0614 13:59:18.326423 20 log.go:36] While running doCompact: MANIFEST removes non-existing table 113984
W0614 13:59:18.326469 20 log.go:36] [Compactor: 0] LOG Compact FAILED with error: MANIFEST removes non-existing table 113988: {span:0xc00ca4c580 compactorId:0 t:{baseLevel:3 targetSz:[0 10485760 10485760 10485760 18157929 181579297 1815792975] fileSz:[67108864 2097152 2097152 2097152 4194304 8388608 16777216]} p:{level:3 score:23.558347129821776 adjusted:26.485386104764444 dropPrefixes:[] t:{baseLevel:3 targetSz:[0 10485760 10485760 10485760 18157929 181579297 1815792975] fileSz:[67108864 2097152 2097152 2097152 4194304 8388608 16777216]}} thisLevel:0xc0000d3140 nextLevel:0xc0000d31a0 top:[0xc0155da240] bot:[0xc00d4a3ec0] thisRange:{left:[0 0 0 0 0 0 0 0 0 0 17 101 98 97 121 46 108 105 115 116 105 110 103 46 110 97 109 101 2 10 99 116 108 0 0 0 0 0 0 0 0] right:[0 0 0 0 0 0 0 0 0 0 17 101 98 97 121 46 108 105 115 116 105 110 103 46 110 97 109 101 2 10 100 101 100 255 255 255 255 255 255 255 255] inf:false size:0} nextRange:{left:[0 0 0 0 0 0 0 0 0 0 17 101 98 97 121 46 108 105 115 116 105 110 103 46 110 97 109 101 2 10 99 101 109 0 0 0 0 0 0 0 0] right:[0 0 0 0 0 0 0 0 0 0 17 101 98 97 121 46 108 105 115 116 105 110 103 46 110 97 109 101 2 10 100 111 119 255 255 255 255 255 255 255 255] inf:false size:0} splits:[] thisSize:2015315 dropPrefixes:[]}
W0614 13:59:18.326555 20 log.go:36] While running doCompact: MANIFEST removes non-existing table 113988
W0614 13:59:18.326541 20 log.go:36] [Compactor: 3] LOG Compact FAILED with error: MANIFEST removes non-existing table 113979: {span:0xc00847e180 compactorId:3 t:{baseLevel:3 targetSz:[0 10485760 10485760 10485760 18157929 181579297 1815792975] fileSz:[67108864 2097152 2097152 2097152 4194304 8388608 16777216]} p:{level:3 score:23.556907653808594 adjusted:26.483767779090424 dropPrefixes:[] t:{baseLevel:3 targetSz:[0 10485760 10485760 10485760 18157929 181579297 1815792975] fileSz:[67108864 2097152 2097152 2097152 4194304 8388608 16777216]}} thisLevel:0xc0000d3140 nextLevel:0xc0000d31a0 top:[0xc00e532000] bot:[0xc00928fd40] thisRange:{left:[0 0 0 0 0 0 0 0 0 0 17 101 98 97 121 46 108 105 115 116 105 110 103 46 110 97 109 101 2 10 83 105 114 0 0 0 0 0 0 0 0] right:[0 0 0 0 0 0 0 0 0 0 17 101 98 97 121 46 108 105 115 116 105 110 103 46 110 97 109 101 2 10 84 97 103 255 255 255 255 255 255 255 255] inf:false size:0} nextRange:{left:[0 0 0 0 0 0 0 0 0 0 17 101 98 97 121 46 108 105 115 116 105 110 103 46 110 97 109 101 2 10 82 101 115 0 0 0 0 0 0 0 0] right:[0 0 0 0 0 0 0 0 0 0 17 101 98 97 121 46 108 105 115 116 105 110 103 46 110 97 109 101 2 10 84 104 101 255 255 255 255 255 255 255 255] inf:false size:0} splits:[] thisSize:2000221 dropPrefixes:[]}
W0614 13:59:18.326577 20 log.go:36] While running doCompact: MANIFEST removes non-existing table 113979
W0614 13:59:18.326925 20 log.go:36] [Compactor: 1] LOG Compact FAILED with error: MANIFEST removes non-existing table 112879: {span:0xc0064b4080 compactorId:1 t:{baseLevel:3 targetSz:[0 10485760 10485760 10485760 18157929 181579297 1815792975] fileSz:[67108864 2097152 2097152 2097152 4194304 8388608 16777216]} p:{level:4 score:1.110077861853078 adjusted:1.116691118230626 dropPrefixes:[] t:{baseLevel:3 targetSz:[0 10485760 10485760 10485760 18157929 181579297 1815792975] fileSz:[67108864 2097152 2097152 2097152 4194304 8388608 16777216]}} thisLevel:0xc0000d31a0 nextLevel:0xc0000d3200 top:[0xc00f389b00] bot:[0xc0155dba40] thisRange:{left:[0 0 0 0 0 0 0 0 0 0 17 101 98 97 121 46 108 105 115 116 105 110 103 46 110 97 109 101 2 10 100 111 120 0 0 0 0 0 0 0 0] right:[0 0 0 0 0 0 0 0 0 0 17 101 98 97 121 46 108 105 115 116 105 110 103 46 110 97 109 101 2 10 101 32 103 255 255 255 255 255 255 255 255] inf:false size:0} nextRange:{left:[0 0 0 0 0 0 0 0 0 0 17 101 98 97 121 46 108 105 115 116 105 110 103 46 110 97 109 101 2 10 100 32 66 0 0 0 0 0 0 0 0] right:[0 0 0 0 0 0 0 0 0 0 17 101 98 97 121 46 108 105 115 116 105 110 103 46 110 97 109 101 2 10 101 32 68 255 255 255 255 255 255 255 255] inf:false size:0} splits:[] thisSize:4005514 dropPrefixes:[]}
W0614 13:59:18.327005 20 log.go:36] While running doCompact: MANIFEST removes non-existing table 112879
W0614 13:59:18.366050 20 log.go:36] [Compactor: 3] LOG Compact FAILED with error: MANIFEST removes non-existing table 113988: {span:0xc00847e380 compactorId:3 t:{baseLevel:3 targetSz:[0 10485760 10485760 10485760 18157929 181579297 1815792975] fileSz:[67108864 2097152 2097152 2097152 4194304 8388608 16777216]} p:{level:3 score:23.558347129821776 adjusted:26.485386104764444 dropPrefixes:[] t:{baseLevel:3 targetSz:[0 10485760 10485760 10485760 18157929 181579297 1815792975] fileSz:[67108864 2097152 2097152 2097152 4194304 8388608 16777216]}} thisLevel:0xc0000d3140 nextLevel:0xc0000d31a0 top:[0xc0155da240] bot:[0xc00d4a3ec0] thisRange:{left:[0 0 0 0 0 0 0 0 0 0 17 101 98 97 121 46 108 105 115 116 105 110 103 46 110 97 109 101 2 10 99 116 108 0 0 0 0 0 0 0 0] right:[0 0 0 0 0 0 0 0 0 0 17 101 98 97 121 46 108 105 115 116 105 110 103 46 110 97 109 101 2 10 100 101 100 255 255 255 255 255 255 255 255] inf:false size:0} nextRange:{left:[0 0 0 0 0 0 0 0 0 0 17 101 98 97 121 46 108 105 115 116 105 110 103 46 110 97 109 101 2 10 99 101 109 0 0 0 0 0 0 0 0] right:[0 0 0 0 0 0 0 0 0 0 17 101 98 97 121 46 108 105 115 116 105 110 103 46 110 97 109 101 2 10 100 111 119 255 255 255 255 255 255 255 255] inf:false size:0} splits:[] thisSize:2015315 dropPrefixes:[]}
W0614 13:59:18.366129 20 log.go:36] While running doCompact: MANIFEST removes non-existing table 113988
W0614 13:59:18.366108 20 log.go:36] [Compactor: 1] LOG Compact FAILED with error: MANIFEST removes non-existing table 113991: {span:0xc0097be300 compactorId:1 t:{baseLevel:3 targetSz:[0 10485760 10485760 10485760 18157929 181579297 1815792975] fileSz:[67108864 2097152 2097152 2097152 4194304 8388608 16777216]} p:{level:3 score:23.36615171432495 adjusted:21.04911062303261 dropPrefixes:[] t:{baseLevel:3 targetSz:[0 10485760 10485760 10485760 18157929 181579297 1815792975] fileSz:[67108864 2097152 2097152 2097152 4194304 8388608 16777216]}} thisLevel:0xc0000d3140 nextLevel:0xc0000d31a0 top:[0xc0147c23c0] bot:[0xc00f389b00] thisRange:{left:[0 0 0 0 0 0 0 0 0 0 17 101 98 97 121 46 108 105 115 116 105 110 103 46 110 97 109 101 2 10 101 32 85 0 0 0 0 0 0 0 0] right:[0 0 0 0 0 0 0 0 0 0 17 101 98 97 121 46 108 105 115 116 105 110 103 46 110 97 109 101 2 10 101 100 32 255 255 255 255 255 255 255 255] inf:false size:0} nextRange:{left:[0 0 0 0 0 0 0 0 0 0 17 101 98 97 121 46 108 105 115 116 105 110 103 46 110 97 109 101 2 10 100 111 120 0 0 0 0 0 0 0 0] right:[0 0 0 0 0 0 0 0 0 0 17 101 98 97 121 46 108 105 115 116 105 110 103 46 110 97 109 101 2 10 101 32 103 255 255 255 255 255 255 255 255] inf:false size:0} splits:[] thisSize:2368028 dropPrefixes:[]}
W0614 13:59:18.366203 20 log.go:36] While running doCompact: MANIFEST removes non-existing table 113991
W0614 13:59:18.366261 20 log.go:36] [Compactor: 0] LOG Compact FAILED with error: MANIFEST removes non-existing table 113979: {span:0xc00847e300 compactorId:0 t:{baseLevel:3 targetSz:[0 10485760 10485760 10485760 18157929 181579297 1815792975] fileSz:[67108864 2097152 2097152 2097152 4194304 8388608 16777216]} p:{level:3 score:23.558347129821776 adjusted:26.485386104764444 dropPrefixes:[] t:{baseLevel:3 targetSz:[0 10485760 10485760 10485760 18157929 181579297 1815792975] fileSz:[67108864 2097152 2097152 2097152 4194304 8388608 16777216]}} thisLevel:0xc0000d3140 nextLevel:0xc0000d31a0 top:[0xc00e532000] bot:[0xc00928fd40] thisRange:{left:[0 0 0 0 0 0 0 0 0 0 17 101 98 97 121 46 108 105 115 116 105 110 103 46 110 97 109 101 2 10 83 105 114 0 0 0 0 0 0 0 0] right:[0 0 0 0 0 0 0 0 0 0 17 101 98 97 121 46 108 105 115 116 105 110 103 46 110 97 109 101 2 10 84 97 103 255 255 255 255 255 255 255 255] inf:false size:0} nextRange:{left:[0 0 0 0 0 0 0 0 0 0 17 101 98 97 121 46 108 105 115 116 105 110 103 46 110 97 109 101 2 10 82 101 115 0 0 0 0 0 0 0 0] right:[0 0 0 0 0 0 0 0 0 0 17 101 98 97 121 46 108 105 115 116 105 110 103 46 110 97 109 101 2 10 84 104 101 255 255 255 255 255 255 255 255] inf:false size:0} splits:[] thisSize:2000221 dropPrefixes:[]}
W0614 13:59:18.366554 20 log.go:36] While running doCompact: MANIFEST removes non-existing table 113979

Looks pretty critical to me.

1 Like