We have noticed that at every restart the number of vlog files increases even if no writes were performed prior to the restart. The restart might be the result of a forced shutdown (preemtible nodes restarting) or a kubernetes probe killing the process.
Badger would replay data on crash but that’s not supposed to create vlog file. I ran a few tests by killing badger locally and I dont see any new vlog files.
Are you using the latest released version of badger? There were two fixes related to replay. If badger would crash during replay, it would replay the entire database on start. These were fixed by
Yes, we are aware of it and that’s why we’re working on separating the write-ahead log with the vlog files. We’re working on it
They are not. Try running this program Go Playground - The Go Programming Language and run watch -n 0.1 "ls -lh ./data" you would see that badger is not creating any new vlog files.
Yes, this could be the reason. I recommend using the latest patch release of badger.
Use the latest version of badger. The older version would go into loop if it crashes on replay.
@christian-roggia please do try the latest badger release and your replay problem should be fixed.
Just out of curiosity, can you show me the logs? I’m looking for the logs which say Replaying file id ... . If you can show me the logs across crashes, that would be perfect and it will help in confirming my suspicion.