Dgraph fails to start on restarts with Kind (Kubernetes)

tvvignesh · October 29, 2020, 11:49am

Just had a look at these:

and the docs here (Use preemptible VMs to run fault-tolerant workloads | Google Kubernetes Engine (GKE) | Google Cloud) say: Do not use Preemptible VMs with stateful Pods because they could violate the at-most-one semantics inherent to StatefulSets and could lead to data loss.

github.com/kubernetes/kubernetes

docs: StatefulSet pod is never evicted from shutdown node

opened 08:26PM - 22 Oct 17 UTC

closed 07:13PM - 31 Dec 22 UTC

at1984z

area/app-lifecycle area/usability area/nodecontroller sig/node area/stateful-apps area/node-lifecycle lifecycle/rotten needs-triage

**Is this a BUG REPORT or FEATURE REQUEST?**: > Uncomment only one, leave it on its own line: > /kind bug > /kind feature **What happened**: 1. StatefulSet at scale 1 is created. 2. The only pod is placed and running on one of 2 worker nodes. 3. The worker node with running pod shuts down and never starts up again. 4. The pod never moved to other node. **What you expected to happen**: 1. The pod would move to other node after expiration of default tolerations "node.alpha.kubernetes.io/notReady:NoExecute for 300s" and "node.alpha.kubernetes.io/unreachable:NoExecute for 300s" 2. "kubectl delete pod pod-on-shutdown-node" would induce the expected movement while the node is down -- it did not happen either. **How to reproduce it (as minimally and precisely as possible)**: 1. Create StatefulSet spec with one container and one replica in, say, sset.yml. 2. Have kubernetes installation with 2 worker nodes. 3. kubectl create -f sset.yml 4. kubectl get pod, to check to see where the only pod is scheduled, say, node N. 5. shutdown node N with "shutdown -h". 6. check to see that the pod did not move to other worker node in 10 minutes after node N halt time. **Anything else we need to know?**: 1. A Deployment behaves as indicated in the "What you expected to happen" section. **Environment**: - Kubernetes version (use `kubectl version`): 1.8.1 - Cloud provider or hardware configuration**: Virtual machines with Vagrant 2.0.0 and VirtualBox 5.1.28-117968 on Intel(R) Xeon(R) CPU E5-2690 v3 24 cores with Ubuntu 16.04 LTS - OS (e.g. from /etc/os-release): Ubuntu 16.04.3 LTS (VM) - Kernel (e.g. `uname -a`): 4.4.0-96-generic (VM) - Install tools: kubeadm 1.8.1-00 - Others: --- Edit: Goal if this issue is to update the documentation and clarify the expected behavior as per: https://github.com/kubernetes/kubernetes/issues/54368#issuecomment-339378597

Wondering what to do.

Topic		Replies	Views
Dgraph crash loop on aws Dgraph	16	1627	June 8, 2020
Alpha node restart failed Dgraph dgraph	12	1587	February 15, 2024
Can I use multiple path for data storage Dgraph kind:question	29	1242	January 28, 2021
DGraph deployment via helm not working anymore Users	4	1621	January 7, 2019
Using Kubernetes - Deploy Documentation	1	1438	August 28, 2020

Dgraph fails to start on restarts with Kind (Kubernetes)

Related topics