Need expand(_all_) like behaviour where schema isn't known beforehand

Users Dgraph
5

seems like your use case is more along the lines of elasticsearch - any new field that comes in just automatically index and be able to search on - this is not the default modus operandi of dgraph. Any new fields ingested will be assigned to a group unindexed until it is told to index them. Indicies are required to run many of the filters necessary to begin a query.

To make it simple: no, there is no built in way to expand(everyPossibleField) without extra work.

Suggestion: assuming you have a structured input, keep the information split into predicates by field and keep them together in one field together in eg: JSON.

eg:

_:newLog <request_time> "2021/01/02-01:02:00" .
_:newLog <service_name> "thanos" .
_:newLog <response_code> "200" .
_:newLog <full_log> "{\"request_time\":\"2021/01/02-01:02:00\",\"service_name\":\"thanos\",\"response_code\":200}" .

then your queries will look like this:

{
  q(func: eq(service_name,"thanos")) @filter(eq(response_code,"200")) {
    full_log
  }
}

that way you will always get ‘all’ of the fields because they will be in full log no matter what you filter by. You can index whatever fields you need to whenever you decide you need to build indicies for them.

edit: and if this works for you, lobby for the JSON type to be added as a feature to the database, as it will make this feel a bit less icky.

1 Like