Thank you so much for the time you have taken to look in to this @dmai .
I was aware that the number of predicates used was really poorly planned, and I have been looking in to addressing that, but aside from using way too much memory, I was wondering why the program was quitting due to memory reasons when the system had plenty of memory available.
The heap profile shown looks to be sampled during the map phase, but I was able to get past that just by increasing resources, but once the bulk loader progressed to the reduce phase it would panic before available memory was actually consumed (at around 100GiB of usage when 210GiB were available).
My import data has been updated to track variable/high cardinality components as facets, and this was very effective in getting memory usage down during the map phase. Usage hovered at around 1GiB for the entirety of that process.
However, when transitioning to the reduce phase, memory consumption increased fairly quickly and plateaued at around 35GiB… granted it got much further than before (89% vs 45%), but it eventually also panicked with out of memory messages even though the process has 180GiB more ram available.
Could this be due to using facets instead of restructuring the data more extensively with additional nodes and properties? I would like to be able to compare both approaches in other areas of usage.
Interestingly some of the final bits of output read as
Finishing stream id: 70876
Finishing stream id: 70877
Finishing stream id: 70878
[16:47:02Z] REDUCE 01h29m57s 88.42% edge_count:328.5M edge_speed:271.7k/sec plist_count:175.1M plist_speed:144.8k/sec. Num Encoding MBs: 0. jemalloc: 2.6 TiB
Finishing stream id: 70879
Which I find interesting since I don’t know that system could even provide virtual memory in that amount.
If this is interesting to you, I captured some heap profiles from pprof which I can pass along, and I can also provide updated data
Thanks again,
Steven