Bulk loader still OOM during reduce phase

Sorry to hear you are having this trouble - just wanted to give my bulkload experience here:

  • ~3 billion triples
  • 4 input rdf files
    • 5-10GiB gzip compressed each
    • pulled from minio
  • 4 output groups
  • 16core 32GiB machine the bulk loader was run on, never got above 25GiB memory.
  • finishes in about 1h20m
  • v21.03.1
  • I think default flags other than the output groups
1 Like