Hi all. I have experienced this about three times now. My macbook’s battery is running out of juice, so, from time to time it just shuts down without no previous alarm if the charger is not connected.
Well, it happens that after I power it up again, the previous “ok” running queries starts to return no results.
See, I cant say for sure that the data is lost, because (and there is a feature that I miss A LOT) I do not know how to ask for the database if there anything in there at all, like a simple “select *”.
This is freaking me out, 'cause I’m moving forward into DGraph at full speed with my project.
Another (partially) related issue is with the “expand(all)”. Why is it such a drama queen? If I do the above query after the crash and no data is found anymore, I would expect just a simple “no results” and not an error.
I see, so the problem is not directly linked to the Dgraph. Right? Crash itself is not done by nor a common activity done by queries, alter or mutation right?
In this situation, in your picture. I believe the Dgraph was not programmed to predict this. It is assumed that the Dgraph instance would be safe on a PC with 99.99% uptime. (PS: I assuming this, speaking for myself)
For you to have your data secured you need to at least give a shutdown command (via HTTP) and in the terminal use “control + c” to finalize the Dgraph correctly. And always make a export as backup.
If you do not do this procedure. Dgraph will keep some locks in files and prevent access to a new instance (in its view). I believe it would be an interesting feature to have a safe process of recovering from bad shutdown in cases like this. But at the moment there is no other way out, I believe.
For me, it does not seem to be server crash. As said your machine ran out of power and hung up subtly.
Who is assuming this is me. I’m just giving you what I know. Perhaps @pawan an better enlighten on some process of recovery from loss of energy.
But if you will have a service that will be public. You have to ensure that the machine will be on 99.99% uptime. And then do the measures I’ve outlined above to make sure the data is not locked or lost.
I believe if you have not “write over” to Dgraph’s folders. Your data is still there. But they’re with Lock. Just the guys to tell you something about it.
By “server crash” I was not meaning “DGraph Server” crash, but “Hardware server” crash instead.
Some of my projects are for customers that held their own datacenters, and they are not always a pretty thing to look at, so this kind of failure can and WILL happen, its just a matter of when.
This is stuff that a relational database is used to, and while sometimes you end up with a corrupted database only recoverable via backup, this is not the case most of the times.
I do spect the same kind of fault tolerance on DGraph, as for me, at least, that is a tremendous no-no.
I see your point about the locks, and have a feeling that you’re right. Lets await for someone drop in with more light to us.
Dgraph is not supposed to lose any data. Everything is written to a write ahead log before returning back to the user and hence Dgraph should recover fine from crashes. What log file are you talking about?
The expand(_all_) not working sometimes is a bug. Can you file a Github issue for it and we can fix it?
Hi Pawan. Sorry, I typed wrong. I meant “LOCK” files. At the last macbook crash down 'cause the battery issue, I deleted the LOCK files inside p, z and zw before restart and the data was ok.
I will let some more crashes happen just to do more tests and will report it back, ok.
Data loss is a blasphemous thing to say, I know it. =]
Hi. After a bunch of more crashes I coudnt see any data loss, nor even the need to delete de LOCK files. I was probably messing up things like a good dumb newbie would do.