Cannot run "Intro to Dgraph" docker image after Docker Desktop update

thefliik · June 25, 2021, 9:39pm

I recently upgraded docker for mac from ~ version 2.5 to version 3.5.1 and I can no longer run the docker image provided in the “Intro to Dgraph” tutorial (both the “docker common” and “docker standalone” images fail with the same error). Before upgrading docker, I could successfully run the “Intro to Dgraph” docker image. If I attempt to run the non-standalone version of the docker image, I can successfully run the “Dgraph zero” image, but when I open a second terminal and run the “Dgraph alpha” image it produces an error and exits:

2021/06/25 21:25:59 1 state.commit 2322 is out of range [1795, 2321]
panic: 1 state.commit 2322 is out of range [1795, 2321]

I’ve tried resetting docker to factory defaults, but the issue persists.

My environment:

MacOs: 10.14.6
Docker Desktop: 3.5.1
Docker Engine: 20.10.7
Docker Compose: 1.29.2
Kubernetes: 1.21.2

Full console printout from the dgraph alpha:

prompt$ docker exec -it dgraph dgraph alpha --cache size-mb=2048 --zero localhost:5080 --security whitelist=0.0.0.0/0
[Sentry] 2021/06/25 21:25:56 Integration installed: ContextifyFrames
[Sentry] 2021/06/25 21:25:56 Integration installed: Environment
[Sentry] 2021/06/25 21:25:56 Integration installed: Modules
[Sentry] 2021/06/25 21:25:56 Integration installed: IgnoreErrors
[Sentry] 2021/06/25 21:25:57 Integration installed: ContextifyFrames
[Sentry] 2021/06/25 21:25:57 Integration installed: Environment
[Sentry] 2021/06/25 21:25:57 Integration installed: Modules
[Sentry] 2021/06/25 21:25:57 Integration installed: IgnoreErrors
I0625 21:25:58.074427      50 sentry_integration.go:48] This instance of Dgraph will send anonymous reports of panics back to Dgraph Labs via Sentry. No confidential information is sent. These reports help improve Dgraph. To opt-out, restart your instance with the --telemetry "sentry=false;" flag. For more info, see https://dgraph.io/docs/howto/#data-handling.
I0625 21:25:58.377740      50 init.go:110] 

Dgraph version   : v21.03.0
Dgraph codename  : rocket
Dgraph SHA-256   : b4e4c77011e2938e9da197395dbce91d0c6ebb83d383b190f5b70201836a773f
Commit SHA-1     : a77bbe8ae
Commit timestamp : 2021-04-07 21:36:38 +0530
Branch           : HEAD
Go version       : go1.16.2
jemalloc enabled : true

For Dgraph official documentation, visit https://dgraph.io/docs.
For discussions about Dgraph     , visit http://discuss.hypermode.com.
For fully-managed Dgraph Cloud   , visit https://dgraph.io/cloud.

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2021 Dgraph Labs, Inc.


I0625 21:25:58.377793      50 run.go:756] x.Config: {PortOffset:0 Limit:normalize-node=10000; mutations-nquad=1000000; max-pending-queries=10000; mutations=allow; query-edge=1000000; disallow-drop=false; query-timeout=0ms; txn-abort-after=5m; max-retries=-1 LimitMutationsNquad:1000000 LimitQueryEdge:1000000 BlockClusterWideDrop:false LimitNormalizeNode:10000 QueryTimeout:0s MaxRetries:-1 GraphQL:extensions=true; poll-interval=1s; lambda-url=; introspection=true; debug=false GraphQLDebug:false}
I0625 21:25:58.378386      50 run.go:757] x.WorkerConfig: {TmpDir:t ExportPath:export Trace:ratio=0.01; jaeger=; datadog= MyAddr: ZeroAddr:[localhost:5080] TLSClientConfig:<nil> TLSServerConfig:<nil> Raft:idx=; group=; learner=false; snapshot-after-entries=10000; snapshot-after-duration=30m; pending-proposals=256 Badger:{Dir: ValueDir: SyncWrites:false NumVersionsToKeep:1 ReadOnly:false Logger:0xc0004f2fd0 Compression:1 InMemory:false MetricsEnabled:true NumGoroutines:8 MemTableSize:67108864 BaseTableSize:2097152 BaseLevelSize:10485760 LevelSizeMultiplier:10 TableSizeMultiplier:2 MaxLevels:7 VLogPercentile:0 ValueThreshold:1048576 NumMemtables:5 BlockSize:4096 BloomFalsePositive:0.01 BlockCacheSize:1395864371 IndexCacheSize:751619276 NumLevelZeroTables:5 NumLevelZeroTablesStall:15 ValueLogFileSize:1073741823 ValueLogMaxEntries:1000000 NumCompactors:4 CompactL0OnClose:false LmaxCompaction:false ZSTDCompressionLevel:0 VerifyValueChecksum:false EncryptionKey:[] EncryptionKeyRotationDuration:240h0m0s BypassLockGuard:false ChecksumVerificationMode:0 DetectConflicts:true NamespaceOffset:-1 managedTxns:false maxBatchCount:0 maxBatchSize:0 maxValueThreshold:0} WhiteListedIPRanges:[{Lower:0.0.0.0 Upper:255.255.255.255}] StrictMutations:false AclEnabled:false HmacSecret:**** AbortOlderThan:5m0s ProposedGroupId:0 StartTime:2021-06-25 21:25:57.490008885 +0000 UTC m=+0.335809081 Ludicrous:enabled=false; concurrency=2000 LudicrousEnabled:false Security:whitelist=0.0.0.0/0; token= EncryptionKey:**** LogRequest:0 HardSync:false Audit:false}
I0625 21:25:58.378616      50 run.go:758] worker.Config: {PostingDir:p WALDir:w MutationsMode:0 AuthToken: HmacSecret:**** AccessJwtTtl:0s RefreshJwtTtl:0s CachePercentage:0,65,35 CacheMb:2048 Audit:<nil> ChangeDataConf:file=; kafka=; sasl_user=; sasl_password=; ca_cert=; client_cert=; client_key=;}
I0625 21:25:58.389042      50 log.go:295] Found file: 1 First Index: 1
I0625 21:25:58.395976      50 storage.go:125] Init Raft Storage with snap: 1795, first: 1796, last: 2321
I0625 21:25:58.397193      50 server_state.go:141] Opening postings BadgerDB with options: {Dir:p ValueDir:p SyncWrites:false NumVersionsToKeep:2147483647 ReadOnly:false Logger:0x32c8038 Compression:1 InMemory:false MetricsEnabled:true NumGoroutines:8 MemTableSize:67108864 BaseTableSize:2097152 BaseLevelSize:10485760 LevelSizeMultiplier:10 TableSizeMultiplier:2 MaxLevels:7 VLogPercentile:0 ValueThreshold:1048576 NumMemtables:5 BlockSize:4096 BloomFalsePositive:0.01 BlockCacheSize:1395864371 IndexCacheSize:751619276 NumLevelZeroTables:5 NumLevelZeroTablesStall:15 ValueLogFileSize:1073741823 ValueLogMaxEntries:1000000 NumCompactors:4 CompactL0OnClose:false LmaxCompaction:false ZSTDCompressionLevel:0 VerifyValueChecksum:false EncryptionKey:[] EncryptionKeyRotationDuration:240h0m0s BypassLockGuard:false ChecksumVerificationMode:0 DetectConflicts:false NamespaceOffset:1 managedTxns:false maxBatchCount:0 maxBatchSize:0 maxValueThreshold:0}
I0625 21:25:58.519555      50 log.go:34] All 4 tables opened in 36ms
I0625 21:25:58.522106      50 log.go:34] Discard stats nextEmptySlot: 0
I0625 21:25:58.522176      50 log.go:34] Set nextTxnTs to 2110
I0625 21:25:58.528880      50 log.go:34] Deleting empty file: p/000005.vlog
I0625 21:25:58.548722      50 groups.go:99] Current Raft Id: 0x1
I0625 21:25:58.549089      50 groups.go:115] Sending member request to Zero: id:1 addr:"localhost:7080" 
I0625 21:25:58.548924      50 worker.go:111] Worker listening at address: [::]:7080
I0625 21:25:58.550155      50 run.go:568] Bringing up GraphQL HTTP API at 0.0.0.0:8080/graphql
I0625 21:25:58.550202      50 run.go:569] Bringing up GraphQL HTTP admin API at 0.0.0.0:8080/admin
I0625 21:25:58.550228      50 run.go:596] gRPC server started.  Listening on port 9080
I0625 21:25:58.550236      50 run.go:597] HTTP server started.  Listening on port 8080
E0625 21:25:58.550259      50 groups.go:1177] Error during SubscribeForUpdates for prefix "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x15dgraph.graphql.schema\x00": Unable to find any servers for group: 1. closer err: <nil>
I0625 21:25:58.650188      50 pool.go:162] CONNECTING to localhost:5080
I0625 21:25:58.655263      50 groups.go:134] Connected to group zero. Assigned group: 0
I0625 21:25:58.655300      50 groups.go:136] Raft Id after connection to Zero: 0x1
I0625 21:25:58.655839      50 draft.go:232] Node ID: 0x1 with GroupID: 1
I0625 21:25:58.656051      50 draft.go:241] RaftContext: id:1 group:1 addr:"localhost:7080" 
I0625 21:25:58.656395      50 node.go:152] Setting raft.Config to: &{ID:1 peers:[] learners:[] ElectionTick:20 HeartbeatTick:1 Storage:0xc00013f3c0 Applied:1795 MaxSizePerMsg:262144 MaxCommittedSizePerReady:67108864 MaxUncommittedEntriesSize:0 MaxInflightMsgs:256 CheckQuorum:false PreVote:true ReadOnlyOption:0 Logger:0x32c8038 DisableProposalForwarding:false}
I0625 21:25:58.674722      50 node.go:310] Found Snapshot.Metadata: {ConfState:{Nodes:[1] Learners:[] XXX_unrecognized:[]} Index:1795 Term:2 XXX_unrecognized:[]}
I0625 21:25:58.674785      50 node.go:321] Found hardstate: {Term:3 Vote:1 Commit:2322 XXX_unrecognized:[]}
I0625 21:25:59.138823      50 node.go:326] Group 1 found 2321 entries
I0625 21:25:59.138970      50 draft.go:1764] Restarting node for group: 1
I0625 21:25:59.139103      50 node.go:189] Setting conf state to nodes:1 
2021/06/25 21:25:59 1 state.commit 2322 is out of range [1795, 2321]
panic: 1 state.commit 2322 is out of range [1795, 2321]

goroutine 153 [running]:
log.Panicf(0x20bb154, 0x2b, 0xc0006387c0, 0x4, 0x4)
	/usr/local/go/src/log/log.go:361 +0xc5
github.com/dgraph-io/dgraph/x.(*ToGlog).Panicf(0x32c8038, 0x20bb154, 0x2b, 0xc0006387c0, 0x4, 0x4)
	/ext-go/1/src/github.com/dgraph-io/dgraph/x/log.go:40 +0x53
go.etcd.io/etcd/raft.(*raft).loadState(0xc0002f0640, 0x3, 0x1, 0x912, 0x0, 0x0, 0x0)
	/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190228193606-a943ad0ee4c9/raft/raft.go:1475 +0x1c4
go.etcd.io/etcd/raft.newRaft(0xc0003b0210, 0x100)
	/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190228193606-a943ad0ee4c9/raft/raft.go:377 +0xd15
go.etcd.io/etcd/raft.RestartNode(0xc0003b0210, 0x1, 0xc00037a7c0)
	/go/pkg/mod/go.etcd.io/etcd@v0.0.0-20190228193606-a943ad0ee4c9/raft/node.go:242 +0x45
github.com/dgraph-io/dgraph/worker.(*node).InitAndStartNode(0xc0004a0b00)
	/ext-go/1/src/github.com/dgraph-io/dgraph/worker/draft.go:1790 +0x545
github.com/dgraph-io/dgraph/worker.StartRaftNodes(0xc00013f3c0, 0xc0005c5f01)
	/ext-go/1/src/github.com/dgraph-io/dgraph/worker/groups.go:154 +0x812
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run.func4(0xc000618210)
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:815 +0x38
created by github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:814 +0x1694
W0625 21:25:59.157061      36 sentry_integration.go:140] unable to read CID from file /tmp/dgraph-alpha-cid-sentry open /tmp/dgraph-alpha-cid-sentry: no such file or directory. Skip
[Sentry] 2021/06/25 21:25:59 Sending fatal event [fd4103a1a348488fa5c918c38db2e65b] to o318308.ingest.sentry.io project: 1805390
[Sentry] 2021/06/25 21:25:59 Buffer flushed successfully.

MichelDiz · June 25, 2021, 11:13pm

This tells me that you have upgraded wrongly. You should at least exported the DB before trying anything.

Have you upgraded Dgraph too? if so, downgrade it.

When the commit state or timestamp gets out of range, this means that the files are corrupted for several reasons. One of them is trying to start a new instance overwriting old files from Dgraph.

Do the following. If the data isn’t important. Do a complete prune in your Docker and start again.

thefliik · June 26, 2021, 1:07am

This tells me that you have upgraded wrongly. You should at least exported the DB before trying anything.

All I’m doing is following the “Tour of Dgraph” instructions (and separately I upgraded Docker Desktop). If the “Tour of Dgraph” instructions haven’t told me to do something, then I haven’t done it.

Have you upgraded Dgraph too? if so, downgrade it.

I just upgraded Docker Desktop.

Do the following. If the data isn’t important. Do a complete prune in your Docker and start again.

I just tried running docker ps to ensure nothing was running, then docker image prune, docker container prune, and docker volume prune. Then I tried running the “docker standalone” image again and still received the same error. And as mentioned, I previously tried resetting docker to factory defaults without success.

Is there anything that the tutorial containers create that would persist across docker resets?

Edit:

I just found the “prune everything” docker command and tried that. Same result as before, unfortunately.

MichelDiz · June 26, 2021, 1:40am

I have installed the newest Docker on my air(M1) and it worked just fine.

Docker Engine v20.10.7
K8s v1.21.2

MichelDiz · June 26, 2021, 1:46am

@thefliik The command has a bind to a path in your OS. -v ~/dgraph:/dgraph make sure that ~/dgraph is clean too. It is in your user path.

thefliik · June 26, 2021, 2:30am

Whew, deleting the ~/dgraph folder did it! Thanks so much for your help!

Topic		Replies	Views
Cannot access Dgraph localhost with docker image Dgraph	4	763	September 9, 2022
Error: Is Dgraph running locally? or reachable? TypeError: Failed to fetch Dgraph kind:question	5	1273	July 22, 2021
Cannot start dgraph after a reboot Dgraph dgraph	12	989	August 24, 2020
Unable to get Dgraph to run using Docker or local installation Dgraph kind:question	15	842	January 20, 2021
Failing to connect Alpha with Zero - Need help in setting the docker cluster setup Dgraph	17	2027	June 23, 2020

Cannot run "Intro to Dgraph" docker image after Docker Desktop update

Related topics