What version of Dgraph are you on?
This might be related to snapshot not happening and the data is kept in memory leading to OOM. Though, I have to see the server logs to confirm. Can you share the logs as well as details about your setup? Dgraph server has an automatic mechanism to abort long pending transactions so that should kick in at some point and free up the memory. Also, what language are you programming in? Maybe you could use one of the official clients and abort/commit your transactions.