Regression: v21.03 crashes on query with orderdesc

miko · May 20, 2021, 1:13am

Report a Dgraph Bug

What version of Dgraph are you using?

All of dgraph/dgraph:master, dgraph/dgraph:v21.03.0 and dgraph/dgraph:v21.03.0-12-g44005bc82 generate error, but dgraph/dgraph:v20.11.2 works OK

Dgraph Version v21.03

$ dgraph version
 
PASTE YOUR RESULTS HERE

Have you tried reproducing the issue with the latest release?

Yes

What is the hardware spec (RAM, OS)?

archlinux 32GB RAM

Steps to reproduce the issue (command/config used to run Dgraph).

I do automatic testing (do new schema, then importing new data - starting from scratch, so no garbage data) which do work for v20.11 but crashed v21.03. Unfortunately I do not have isolated small test case, but hopefully following description is enough.

Following query crashes v21.03:

queryGardenTicket(func: uid(GardenTicketRoot), orderdesc: GardenTicket.created_at) @cascade(GardenTicket.user) {
  uid
}

while the same query without orderdesc works OK:

queryGardenTicket(func: uid(GardenTicketRoot)) @cascade(GardenTicket.user) {
    uid
}

when there is a record as a result.
There is no crash when there is no result, or when using v20.11

Expected behaviour and actual result.

The dgraph alpha crashes with the following stack trace:

dgraph_1         | 2021/05/20 00:52:25 1 0
dgraph_1         | github.com/dgraph-io/dgraph/x.AssertTruef
dgraph_1         | 	/ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:107
dgraph_1         | github.com/dgraph-io/dgraph/worker.intersectBucket
dgraph_1         | 	/ext-go/1/src/github.com/dgraph-io/dgraph/worker/sort.go:684
dgraph_1         | github.com/dgraph-io/dgraph/worker.sortWithIndex
dgraph_1         | 	/ext-go/1/src/github.com/dgraph-io/dgraph/worker/sort.go:286
dgraph_1         | github.com/dgraph-io/dgraph/worker.processSort.func2
dgraph_1         | 	/ext-go/1/src/github.com/dgraph-io/dgraph/worker/sort.go:515
dgraph_1         | runtime.goexit
dgraph_1         | 	/usr/local/go/src/runtime/asm_amd64.s:1371

Expected result is no crash, as it worked for v20.11

chewxy · May 20, 2021, 3:33am

Do you have a schema for me to reproduce?

miko · May 20, 2021, 12:28pm

Yes, I managed to extract the case. Graphql Schema:

enum GardenTicketStatus {
  new
  reserved
  active
  cancelled
  past
  unavailable
  error
}

"User object"
type User @secret(field:"password") 
@auth(
  query:{ or:[
   {rule: "{$role_admin:{eq:\"true\"}}" },
   {rule: "{$internal:{eq:\"true\"}}" },
   {rule:
    """
      query($USER: String!) {
        queryUser(filter:{login:{eq:$USER}}) {
          login
        }
      }
    """
   }
  ]},
  update:{ or:[
   {rule: "{$role_admin:{eq:\"true\"}}" },
   {rule: "{$internal:{eq:\"true\"}}" },
   {rule:
    """
      query($USER: String!) {
        queryUser(filter:{login:{eq:$USER}}) {
          login
        }
      }
    """
   }
  ]},
  delete:{ or:[
   {rule: "{$role_admin:{eq:\"true\"}}" },
   {rule: "{$internal:{eq:\"true\"}}" },
  ]}
)
{
  login: String! @id @search(by: [hash])

  first_name: String! @search(by: [hash])
  last_name: String! @search(by: [hash])
  full_name: String @lambda
  mobile_phone: String @search(by: [hash])

  created_at: DateTime! @search
  modified_at: DateTime! @search

  garden_tickets: [GardenTicket]
}
type GardenTicket @withSubscription
 @auth(
  query:{ or:[
   {rule:
    """
      query($USER: String!) {
        queryGardenTicket @cascade(fields:["user"]){
          user (filter:{login:{eq:$USER}}) {login}
        }
      }
    """
   },
   {rule: "{$role_admin:{eq:\"true\"}}" }
  ]}
)

 {
  id: ID!
  date: String!
  start_hour: String! @search(by: [hash])
  end_hour: String! @search(by: [hash])
  valid_from: DateTime! @search
  valid_to: DateTime! @search
  created_at: DateTime! @search
  user: User! @hasInverse(field:garden_tickets)
  status: GardenTicketStatus! @search(by: [hash])
}

# Dgraph.Authorization {"VerificationKey":"secret","Header":"X-Auth-Token","Namespace":"mydomain.com","Algo":"HS256","Audience":["garden"]}

Sample data:

mutation ($now:DateTime!="2021-05-05T10:00:00Z") {

  addUser(input:[
    {login:"admin",password:"admin1",
      created_at:"2021-01-01T00:00:00Z",
      modified_at:"2021-01-01T00:00:00Z",
      first_name:"Admin",
      last_name:"Admin",
    },
    {login:"john",password:"secret",
      created_at:"2021-01-01T00:00:00Z",
      modified_at:"2021-01-01T00:00:00Z",
      first_name:"John",
      last_name:"Doe",
    },
  ]) {numUids}

  addGardenTicket(input:{
    date:"2021-05-05",
    start_hour:"12:00",
    end_hour:"13:00",
    status:reserved,
    created_at:$now,
    valid_to:$now,
    valid_from:$now,
    user:{login:"john"}
  }) {
    numUids
    gardenTicket {id date created_at status start_hour user {login first_name last_name}}
  }

}

Failing DQL query:

 query {
   queryGardenTicket(func: uid(GardenTicketRoot), orderdesc: GardenTicket.created_at) @cascade(GardenTicket.user) {
   uid
    GardenTicket.user  {uid User.login}
   }
   GardenTicketRoot as var(func: uid(GardenTicket_6))
   GardenTicket_6 as var(func: type(GardenTicket)) @filter((eq(GardenTicket.status, "new", "reserved", "active") AND ge(GardenTicket.valid_to, "2020-05-20T02:28:52+02:00")))
   var(func: uid(GardenTicketRoot)) {
     User_2 as GardenTicket.user @filter(eq(User.login, "john"))
   }
   User_1 as var(func: uid(User_2)) @filter(uid(User_Auth5))
   var(func: uid(User_1)) {
    User.email
   }
   User_Auth5 as var(func: uid(User_2)) @filter(eq(User.login, "john")) @cascade {
     User.login : User.login
   }   
 }

Not failing DQL query:

 query {
   queryGardenTicket(func: uid(GardenTicketRoot))  @cascade(GardenTicket.user) {
   uid
    GardenTicket.user  {uid User.login}
   }
   GardenTicketRoot as var(func: uid(GardenTicket_6))
   GardenTicket_6 as var(func: type(GardenTicket)) @filter((eq(GardenTicket.status, "new", "reserved", "active") AND ge(GardenTicket.valid_to, "2020-05-20T02:28:52+02:00")))
   var(func: uid(GardenTicketRoot)) {
     User_2 as GardenTicket.user @filter(eq(User.login, "john"))
   }
   User_1 as var(func: uid(User_2)) @filter(uid(User_Auth5))
   var(func: uid(User_1)) {
    User.email
   }
   User_Auth5 as var(func: uid(User_2)) @filter(eq(User.login, "john")) @cascade {
     User.login : User.login
   }   
 }

Note: I am using JWT with payload "mydomain.com": {"role_admin":"true"} so @auth rule is satisfied. Not sure if it matters.

chewxy · May 21, 2021, 1:30am

I have accepted this as a bug. @hardik fyi

nick · May 23, 2021, 2:37am

I ran into this same issue, and came up with a fairly minimal example to reproduce the crash:

schema

<category.name>: string .
<category.item>: [uid] .
<item.name>: string @index(exact) .

mutation

{
  set {
    _:category <category.name> "category" .
    _:item1 <item.name> "item 1" .
    _:item2 <item.name> "item 2" .
    _:category <category.item> _:item1 .
    _:category <category.item> _:item2 .
  }
}

query

query ItemsInCategory(
  $categoryUID: string
) {
  itemsInCategory(func: uid($categoryUID)) @cascade {
    category.item (orderdesc: item.name) {
      uid
    }
  }
}

To prevent the crash, you can

remove @cascade from the query
remove (orderdesc: item.name) from the query
remove @index(exact) from item.name in the schema

I hope this might help anyone looking into solving this issue.

doude · May 27, 2021, 3:12pm

I also run into the same issue with that minimal graphql setup:

schema:

type Foo {
  id:   ID!
  date: DateTime! @search(by: [hour])
  bar:  Bar!      @hasInverse(field: foo)
}

type Bar {
  id:  ID!
  foo: Foo! @hasInverse(field: bar)
}

Inject some data:

mutation addFooAndBar {
  addBar(input: {foo: {date: "2021-05-27T17:04:30Z"}}) {
    bar {
      id
      foo {
        id
        date
      }
    }
    numUids
  }
}

query that crash alpha:

query MyQuery {
  queryFoo(order: {asc: date})@cascade(fields:["bar"]) {
    bar {
      id
    }
  }
}

I used the last standalone docker image 21.03:

Dgraph version   : v21.03.0
Dgraph codename  : rocket
Dgraph SHA-256   : b4e4c77011e2938e9da197395dbce91d0c6ebb83d383b190f5b70201836a773f
Commit SHA-1     : a77bbe8ae
Commit timestamp : 2021-04-07 21:36:38 +0530
Branch           : HEAD
Go version       : go1.16.2
jemalloc enabled : true

For Dgraph official documentation, visit https://dgraph.io/docs.
For discussions about Dgraph     , visit http://discuss.hypermode.com.
For fully-managed Dgraph Cloud   , visit https://dgraph.io/cloud.

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2021 Dgraph Labs, Inc.

doude · June 3, 2021, 10:40am

@chewxy any update on that one?
Can we follow the bug you opened on a bug tracker?

I tried that on https://cloud.dgraph.io/ as it moved on 21.03 and the error seems to be there too.

I got a 502 error with the failing query:

$ curl -v 'https://withered-wave.eu-central-1.aws.cloud.dgraph.io/graphql' \
  -X POST \
  -H 'content-type: application/json' \
  --data '{
    "query": "{ queryFoo(order: {asc: date})@cascade(fields:[\"bar\"]) { bar { id } } }"
  }'
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 3.65.76.212:443...
* Connected to withered-wave.eu-central-1.aws.cloud.dgraph.io (3.65.76.212) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:

*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.eu-central-1.aws.cloud.dgraph.io
*  start date: Apr 14 12:35:08 2021 GMT
*  expire date: Jul 13 12:35:08 2021 GMT
*  subjectAltName: host "withered-wave.eu-central-1.aws.cloud.dgraph.io" matched cert's "*.eu-central-1.aws.cloud.dgraph.io"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55afe4163970)
> POST /graphql HTTP/2
> Host: withered-wave.eu-central-1.aws.cloud.dgraph.io
> user-agent: curl/7.77.0
> accept: */*
> content-type: application/json
> content-length: 94
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* We are completely uploaded and fine
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 502
< server: Caddy
< content-length: 0
< date: Fri, 04 Jun 2021 07:15:46 GMT
<
* Connection #0 to host withered-wave.eu-central-1.aws.cloud.dgraph.io left intact

a working query returns a 200:

$ curl -v 'https://withered-wave.eu-central-1.aws.cloud.dgraph.io/graphql' \
  -X POST \
  -H 'content-type: application/json' \
  --data '{
    "query": "{ queryFoo(order: {asc: date}) { bar { id } } }"
  }'
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 3.64.168.155:443...
* Connected to withered-wave.eu-central-1.aws.cloud.dgraph.io (3.64.168.155) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.eu-central-1.aws.cloud.dgraph.io
*  start date: Apr 14 12:35:08 2021 GMT
*  expire date: Jul 13 12:35:08 2021 GMT
*  subjectAltName: host "withered-wave.eu-central-1.aws.cloud.dgraph.io" matched cert's "*.eu-central-1.aws.cloud.dgraph.io"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55ffa9b68970)
> POST /graphql HTTP/2
> Host: withered-wave.eu-central-1.aws.cloud.dgraph.io
> user-agent: curl/7.77.0
> accept: */*
> content-type: application/json
> content-length: 68
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* We are completely uploaded and fine
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200
< access-control-allow-credentials: true
< access-control-allow-headers: X-Dgraph-AccessToken, X-Dgraph-AuthToken, Content-Type, Content-Length, Accept-Encoding, Cache-Control, X-CSRF-Token, X-Auth-Token, X-Requested-With, sentry-trace, DG-Auth
< access-control-allow-methods: POST, OPTIONS
< access-control-allow-origin: *
< content-type: application/json
< date: Fri, 04 Jun 2021 07:17:54 GMT
< graphql-toucheduids: 3
< server: Caddy
< content-length: 437
<
* Connection #0 to host withered-wave.eu-central-1.aws.cloud.dgraph.io left intact
{"data":{"queryFoo":[{"bar":{"id":"0x9c4f5c1"}}]},"extensions":{"touched_uids":3,"tracing":{"version":1,"startTime":"2021-06-04T07:17:54.147804484Z","endTime":"2021-06-04T07:17:54.152054709Z","duration":4250235,"execution":{"resolvers":[{"path":["queryFoo"],"parentType":"Query","fieldName":"queryFoo","returnType":"[Foo]","startOffset":105135,"duration":4140283,"dgraph":[{"label":"query","startOffset":143985,"duration":4099697}]}]}}}}%

chewxy · June 7, 2021, 12:19am

Hi @doude there’s no progress on this yet. @hardik can give a clearer picture why

miko · June 7, 2021, 7:25am

Isn’t this kind of blocker for 21.03? You can kill the server with a simple query Any suggested workaround?

nick · June 13, 2021, 5:46pm

@hardik Any word on this?

hardik · June 14, 2021, 7:29am

This issue has been fixed on Release/21.03 branch, so can you guys try it out and see if it helps? Also the fix would be available in 21.03.1 and will be patched in cloud soon.
cc: @minhaj

miko · June 14, 2021, 9:27am

Thanks! Is there a docker image released with this patch?
Edit: works with dgraph/dgraph:master

nick · June 15, 2021, 3:45pm

Works for me, thanks!

Topic		Replies	Views
Between() in query causes alpha crash Dgraph status:accepted , kind:bug , ticket:created	9	691	October 15, 2020
Critical bug in v21.12 permanently crashloops whole groups Dgraph status:accepted , ticket:created	14	2932	May 19, 2023
Dgraph v21.03 - Superflag JSON config crashes Alpha Dgraph status:accepted , kind:bug , ticket:created	3	655	April 9, 2021
Sorting on multiple predicates broken on 20.11.1 Dgraph status:accepted , kind:bug , ticket:created	5	536	February 23, 2021
Mutations can break queries on unrelated nodes Dgraph dgraph , area:mutations	4	651	May 13, 2020