Problem with dgraph bulk -r command

mrjn · May 15, 2018, 1:31am

If there’s a significant slowdown, it could be:

You were running some other program, which interfered with the bulk loader. This could be a download which would affect the disk throughput, hence affecting the loader.
There’s something about having 2000 files, which might not be fitting well with the loader. We haven’t tested with a directory with so many RDF files; I don’t see why that could be a problem; but who knows.

I’d say check for 1. Retry, and see if you still see that issue. If you still see it, you could try doing a cpu profiling, if you know how to do that (using http): Profiling Go Programs - The Go Programming Language

Otherwise, you could try to merge these files to drop their number. Merging could be done with linux + bash, via zcat, and gzip.

Bulk loader is really meant to be used in one go.

Update

I’m looking at the code,

github.com

dgraph-io/dgraph/blob/e896c17e434f8bdedfa7578f61ba239e9e9059eb/dgraph/cmd/bulk/loader.go#L235


      
          		}
          	}
          
          	if len(readers) == 0 {
          		fmt.Println("No rdf files found.")
          		os.Exit(1)
          	}
          
          	thr := x.NewThrottle(ld.opt.NumGoroutines)
          	for _, r := range readers {
          		thr.Start()
          		go func(r *bufio.Reader) {
          			defer thr.Done()
          			for {
          				chunkBuf, err := readChunk(r)
          				if err == io.EOF {
          					if chunkBuf.Len() != 0 {
          						ld.rdfChunkCh <- chunkBuf
          					}
          					break
          				}

And I don’t really see anything here, which won’t work well with increased number of files. In fact, we’re ensuring that we don’t create as many goroutines as the number of files, by using a throttle. So, cpu performance should be the same as one file.

Topic		Replies	Views
Some mistakes when running dgraph bulk Users	7	623	June 27, 2018
Data Ingestion very slow Users	6	1093	October 25, 2018
Bulk load fails Dgraph	2	450	December 16, 2019
Bulk loader becomes slow when memory gets full Users	20	2171	December 17, 2017
Bulk Loader performance Dgraph kind:question , dgraph	5	488	February 1, 2021

Problem with dgraph bulk -r command

Update

Related topics