To live load from MinIo,Error : All access to this bucket has been disabled

Users Dgraph
,
1

I used Live to import data from Minio, but it didn’t work
I’ve set the environment variable

MINIO_ACCESS_KEY
MINIO_SECRET_KEY

Dgraph:v21.03.0

Command:

dgraph live -a 127.0.0.1:9580 -z 127.0.0.1:5580 -f minio://192.168.31.10:3080/minio/dgraph/data/movie/21million.rdf.gz?secure=false -s minio://192.168.31.10:3080/minio/dgraph/data/movie/movie.schema?secure=false

Error:

root@server13:/home/user/v21.3.0/dgraph# dgraph live -a 127.0.0.1:9580 -z 127.0.0.1:5580 -f minio://192.168.31.10:3080/minio/dgraph/data/movie/21million.rdf.gz?secure=false -s minio://192.168.31.10:3080/minio/dgraph/data/movie/movie.schema?secure=false
I0426 14:34:17.806668   35427 init.go:110]

Dgraph version   : v21.03.0
Dgraph codename  : rocket-mod
Dgraph SHA-256   : 7d5902a16e3169b96d9ba8246ffd769c80eba93f9bdff6cfff52a176da7b4ac4
Commit SHA-1     : a77bbe8ae
Commit timestamp : 2021-04-07 21:36:38 +0530
Branch           : heads/v21.03.0
Go version       : go1.16.2
jemalloc enabled : true

For Dgraph official documentation, visit https://dgraph.io/docs.
For discussions about Dgraph     , visit http://discuss.hypermode.com.
For fully-managed Dgraph Cloud   , visit https://dgraph.io/cloud.

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2021 Dgraph Labs, Inc.



Running transaction with dgraph endpoint: 127.0.0.1:9580

Processing schema file "minio://192.168.31.10:3080/minio/dgraph/data/movie/movie.schema?secure=false"
E0426 14:34:17.816828   35427 run.go:257] Error while reading file ReadAll: All access to this bucket has been disabled.
2021/04/26 14:34:17 All access to this bucket has been disabled.
Error while reading file
github.com/dgraph-io/dgraph/x.Checkf
        /home/user/v21.3.0/dgraph/x/error.go:51
github.com/dgraph-io/dgraph/dgraph/cmd/live.(*loader).processSchemaFile
        /home/user/v21.3.0/dgraph/dgraph/cmd/live/run.go:258
github.com/dgraph-io/dgraph/dgraph/cmd/live.run
        /home/user/v21.3.0/dgraph/dgraph/cmd/live/run.go:800
github.com/dgraph-io/dgraph/dgraph/cmd/live.init.0.func1
        /home/user/v21.3.0/dgraph/dgraph/cmd/live/run.go:134
github.com/spf13/cobra.(*Command).execute
        /home/user/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
        /home/user/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
        /home/user/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
        /home/user/v21.3.0/dgraph/dgraph/cmd/root.go:78
main.main
        /home/user/v21.3.0/dgraph/dgraph/main.go:99
runtime.main
        /snap/go/7221/src/runtime/proc.go:225
runtime.goexit
        /snap/go/7221/src/runtime/asm_amd64.s:1371
2

@joaquin can you help?

3

@lee.chen @MichelDiz For the MinIO service, is it really bound to 3080 port and not the default 9000 port? Is the bucket named minio, with folder path of of data/movie/movie?

I was able to get this working locally using docker-compose.

My steps (running from bash):

## bring up test environment
docker-compose up -d

## using mc command line tool, create a bucket named "movie"
mc alias set local http://localhost:9000 accesskey secretkey
mc mb local/movie

## download files locally
PREFIX=https://github.com/dgraph-io/benchmarks/raw/master/data/
FILES=(21million.schema 21million.rdf.gz)
for FILE in ${FILES[*]}; do 
  curl --silent --location --remote-name $PREFIX/$FILE
done

## upload files to bucket
mc cp 21million.rdf.gz  21million.schema local/movie

## do live loader from alpha1 
## NOTE: This could could be any container running dgraph v21.03.0
##       on the same netowrk.
docker exec -t alpha1 dgraph live \
  --alpha ${DGRAPH_ALPHA}:9080 \
  --zero ${DGRAPH_ZERO}:7080 \
  --schema minio://minio:9000/movie/21million.schema?secure=false \
  --files minio://minio:9000/movie/21million.rdf.gz?secure=false

My docker-compose.yaml file showing configuration of Dgraph Alpha is the following:

version: "3.5"
services:
  zero1:
    image: dgraph/dgraph:v21.03.0
    command: dgraph zero --my=zero1:5080 --replicas 1 --raft idx=1
    container_name: zero1

  alpha1:
    image: dgraph/dgraph:v21.03.0
    environment:
      MINIO_ACCESS_KEY: accesskey
      MINIO_SECRET_KEY: secretkey
      DGRAPH_ALPHA_SECURITY: whitelist=10.0.0.0/8,172.0.0.0/8,192.168.0.0/16
    ports:
      - 8080:8080
      - 9080:9080
    command: dgraph alpha --my=alpha1:7080 --zero=zero1:5080
    container_name: alpha1

  minio:
    image: minio/minio:RELEASE.2021-04-22T15-44-28Z
    environment:
      MINIO_ACCESS_KEY: accesskey
      MINIO_SECRET_KEY: secretkey
    ports:
      - 9000:9000
    command: server /data
    container_name: mini

This is my UI once deployed. I noticed that minio web ui adds minio/ to the http path, which is not part of the actually path (as it’s an artifact of their web service only):

minio
minio894×528 83.2 KB

1 Like
4

I didn’t start Minio in docker-compose, so I can confirm that the port is correct

image
image880×574 17.8 KB

5

The docker-compose was just for example purposes, to show that the functionality works. Was this useful, are you still stuck?

6

@lee.chen you are using the web browser URL /minio/ but you cannot use that when you are accessing objects not via the UI. Take out the /minio/ from your URL givien to the live loader.