I also stumbled on this problem and same as you used the docs to come to the same conclusion.
But after some trial and error and digging it seems to be that
DGRAPH_ALPHA_GRAPHQL: 'lambda-url=http://dgraph_lambda:8686/graphql-worker'
Seems to be the correct input to setup the lambda-url for a alpha server using docker compose