I'd like to build my own Graph Database, where should I start?

Graph isn’t complicated. So you can do this. What is hard is horizontal distribution(which what Dgraph does).

You should start with “Graph Theory”.

However I wouldn’t know what to recommend. You can start anyway, it doesn’t have a secret or recipe. Graph is just math.