Is it possible that alpha stop works when there is no disk space for logs?

Users Dgraph
1

Hi guys.

I’m running my dGraph (v1.0.11) cluster in docker containers on three servers - 3 alphas, 3 zeros (8xcore, 64GB RAM).
Each servers has two ssd disks - one 25 GB and 300 GB.
This morning one of alphas stops to response (may be all cluster). When I checked logs, I found this line as last one in logs:

log: exiting because of error: write /tmp/dgraph.3.root.log.INFO.20190124-015534.1: no space left on device

Docker logs are stored at smaller one of disks.
When I removed old container (also erased logs) and started new one, everything starts to work.

Is it possible that alpha stop works when there is no disk space for logs? Is this bug?

Thanks,
Roman

2

Exiting when there’s no space to write logs is what Google’s glog logging library does:

You can provide sufficient disk space for the log files. If you’re handling logs a different way and don’t need the log files that are written by glog, then you can use the flag --logtostderr, which will only log to stderr and not to files. You can alternatively set --log_dir to somewhere else other than /tmp, the default location.

2 Likes