Does anyone else that uses the dbt dagster library get excep dagster #ask-community

Does anyone else that uses the dbt-dagster library...

Paul Swithers

06/28/2022, 4:42 PM

Does anyone else that uses the dbt-dagster library get exceptions like this: "Pickling client objects is explicitly not supported. Clients have non-trivial state that is local and unpickleable."? What I have found is that this indicates that an upstream model is missing. If I take the name of the model that produces this exception and build it using the dbt CLI, I get a helpful error that tells me which model is missing. This seems consistent with this outdated discussion of a similar exception message from dbt v0.15. Meanwhile, we are running with dbt v1.0.8 and dagster v0.14.17. I am not questioning the exception. I am questioning the "pickling" message. Why do I get a "pickling" message running in dbt-dagster, but a much more useful message running with the dbt CLI? Is this something that dbt Labs can address? If I advance to a more recent version will that resolve the messaging issue?

owen

06/28/2022, 5:39 PM

hi @Paul Swithers ! we've gotten other reports of this issue, and this seems to be a bug with running dbt with the

--log-format json

flag (which the dagster-dbt library does to make it easier for us to parse out the log messages). This is something that will need to be fixed from their end (this bug is still present in the most recent version of dbt as far as I can tell). We're working on a temporary workaround from our side that will hopefully make it into this week's release

👍 2

owen

06/28/2022, 10:51 PM

hey again @Paul Swithers, I've dug into this a bit more, and wasn't able to exactly reproduce the error that you were seeing (although I got something similar). What I was trying was creating two models:

a.sql

select 1

and

b.sql

select * from { ref("a") }

. Then, I just ran

dbt --log-format json run --select b

(so b will select from a table that doesn't yet exist). This gave me a JSON serialization error (rather than a pickle error). I've opened up a PR on dbt's end to fix that issue (https://github.com/dbt-labs/dbt-bigquery/pull/207), but I just want to make sure that this PR will actually fix the issue you're seeing. Is there anything you can think of that would be different about your setup that's leading you to hit a different error?

Paul Swithers

06/29/2022, 12:57 PM

I configure a dagster resource using the dbt_cli_resource from the dagster_dbt package and run dbt commands using the "cli" function of the resulting dbt_cli_resource object.

Paul Swithers

06/30/2022, 5:59 PM

@owen I had a chance to mimic your test and got the pickling error from the command line.

Copy code

dbt --log-format json run --select b

Copy code

{"code": "Z028", "data": {"msg": "Pickling client objects is explicitly not supported."}, "invocation_id": "b9a3c263-5888-43b6-b248-5177908ed7e6", "level": "error", "log_version": 1, "msg": "\u001b[33mPickling client objects is explicitly not supported.\u001b[0m", "node_info": {}, "pid": 93681, "thread_name": "MainThread", "ts": "2022-06-30T17:54:36.030822Z", "type": "log_line"}
{"code": "Z029", "data": {"msg": "Clients have non-trivial state that is local and unpickleable."}, "invocation_id": "b9a3c263-5888-43b6-b248-5177908ed7e6", "level": "error", "log_version": 1, "msg": "Clients have non-trivial state that is local and unpickleable.", "node_info": {}, "pid": 93681, "thread_name": "MainThread", "ts": "2022-06-30T17:54:36.030988Z", "type": "log_line"}

You were right. There must be something about our set up that results in pickling exception messages where you get serialization messages. Could it be that our project is built around BigQuery? Either way, the correct message should be this one: "404 Not found: Table cbh-paul-swithers:dbt_paul_swithers.a was not found in location US". I got that when I ran this:

Copy code

dbt run --select b

49 Views

Open in Slack

Previous Next