Hello Dgraphers,
I’m busy building a pipeline to move data from Athena to new Dgraph clusters using the bulk loader, but I’d like to be able to control the bulk loader programmatically from inside Go.
Are there plans to make the bulk loader available as a package? And if not, would it be something you’d accept a PR for if my team put one together?
The bulk loader is actually available as a package: import "github.com/dgraph-io/cmd/bulk" . The datatypes have mostly unexported fields though. You are looking for the bulk.Bulk datatype (which is a x.SubCommand) to manage your bulk imports.
Bear in mind if you do use Viper/Cobra in your application, the package bulk will have some things that mess with the global state. It may be a bit tricky, but definitely doable.
Er, I guess I phrased that badly. I saw the "github.com/dgraph-io/dgraph/dgraph/cmd/bulk" package, but I want to export more things from it so that I have more options than just calling the subcommand.
For example, I might have RDF data in memory and would like to use loader code to convert it to the Dgraph filesystem directly.