Metric dgraph_max_assigned_ts alert

Users Dgraph
1

What I want to do

The synchronization timestamp status of the Dgraph instance node is abnormal. The following indicator monitoring often shows warnings with a difference of more than 1000. What is causing this?

max(dgraph_max_assigned_ts )  - min(dgraph_max_assigned_ts)

image
image2880×1422 308 KB

What I did

Deployment configuration:
Three zeros and three alphas are deployed in K8s using default startup parameters.
–replicas Set to 3
Alarm configuration comes from

Dgraph metadata

dgraph version 
Dgraph version   : v20.11.2
Dgraph codename  : tchalla-2
Commit timestamp : 2021-02-23 13:07:17 +0530
Branch           : HEAD
Go version       : go1.15.5
jemalloc enabled : true
2

Indicator data at that time:

image
image2742×1554 438 KB

image
image2742×1460 478 KB

image
image2752×1136 374 KB

3

I would like to ask, if this alarm is not handled, will there be any problems?

4

Hi, @chewxy
Please help look at the reasons.

5

It’s a measure of intragroup latency. If you have a group of instances {A, B, C, D, E, F}, the one with the highest maxAssignedTS is usually the leader. It simply tells you how much behind a follower is.

So your alphas are about 2 seconds out of sync with each other in the cluster. Queries could be blocked while waiting for other alphas to catch up.

6

Well, what causes this out of sync? Because it is normal for me to view the indicator data here, each alpha node will receive the client’s request and get the latest timestamp.