Memory issue during stream operation

ibrahim · March 3, 2021, 1:26pm

@bgokden can you try to run badger with jemalloc?
You will have to install jemalloc locally. Please see this

dgraph-io/dgraph/blob/c4c3248db7a2a914171850430953c1890e3d7b75/dgraph/Makefile#L92-L108


jemalloc:
	@if [ -z "$(HAS_JEMALLOC)" ] ; then \
		mkdir -p /tmp/jemalloc-temp && cd /tmp/jemalloc-temp ; \
		echo "Downloading jemalloc" ; \
		curl -s -L ${JEMALLOC_URL} -o jemalloc.tar.bz2 ; \
		tar xjf ./jemalloc.tar.bz2 ; \
		cd jemalloc-5.2.1 ; \
		./configure --with-jemalloc-prefix='je_' --with-malloc-conf='background_thread:true,metadata_thp:auto'; \
		make ; \
		if [ "$(USER_ID)" = "0" ]; then \
			make install ; \
		else \
			echo "==== Need sudo access to install jemalloc" ; \
			sudo make install ; \
		fi \
	fi

And then, you’ll have to build your go program with jemalloc. To do that, run go build -tags=jemalloc ... .

You should see a significant improvement in badger’s memory utilization with jemalloc

Topic		Replies	Views
Provide simple option for limiting total memory usage Issues badger , kind:enhancement , status:accepted , area:documentation	19	2085	May 1, 2022
Why is ristretto using so much memory? Ristretto investigate , ristretto , priority:p2 , status:needs-attenti	17	2844	October 12, 2020
Indefinite memory growth as BadgerDB grows, unaccounted for by golang pprof Badger	2	834	November 7, 2021
Badger performance and memory usage recommendations Badger	9	5389	June 27, 2020
Clarify "Memory usage" section in readme Issues badger , status:accepted , priority:p1 , area:documentation	11	1204	May 18, 2020

Memory issue during stream operation

Related topics