There doesn’t seem to be anything wrong with the timestamps. Here’s the timeline:
- First backup does a backup up to timestamp 21602.
- The query afterwards has a timestamp of 21603. This makes perfect sense.
- Second backup does a backup of the data changed between timestamps 21602 and 21636. Since the difference is pretty small, the resulting backup is expected to be small. The only exception is if your transactions were huge but that doesn’t seem to be the case.
- The strange part happens here. After you run the query, the timestamp is 21637, which is the next timestamp. So it appears that the timestamp the backup received is indeed correct. The strange part is that you said a lot of changes were made between the first and second backups but neither the backup nor Dgraph itself agrees with this.
Just a few more questions to try to get to the bottom of this.
- How big are the transactions sent to Dgraph? Are you sending each change in a separate transaction or batching multiple changes and then writing them to Dgraph?
- Is your code successfully committing the transactions?
- Are you handling aborted transactions that must be retried correctly? If there’s conflicts in the transactions and it cannot be committed, the timestamp will not increase. Your code should retry the mutation(s).
- What’s the rate at which you send data to Dgraph? If it’s variable this could just mean there weren’t many changes between the two points. do you have a mechanism to check how much data was supposed to be added between the two backup points?
After looking at this, I don’t think the issue is with backups at all because the timestamps that the backup received seem correct based on the timestamps of the queries done right after. Either there weren’t any changes made between the two backups or the mutations are not succeeding for some reason.