Does anyone else that uses the dbt dagster library get excep

Question

Does anyone else that uses the dbt dagster library get exceptions like this Pickling client objects is explicitly not supported Clients have non trivial state that is local and unpickleable What I have found is that this indicates that an upstream model is missing If I take the name of the model that produces this exception and build it using the dbt CLI I get a helpful error that tells me which model is missing This seems consistent with this outdated <https github com dbt labs dbt core issues 2110|discussion> of a similar exception message from dbt v0 15 Meanwhile we are running with dbt v1 0 8 and dagster v0 14 17 I am not questioning the exception I am questioning the pickling message Why do I get a pickling message running in dbt dagster but a much more useful message running with the dbt CLI Is this something that dbt Labs can address If I advance to a more recent version will that resolve the messaging issue

owen · Answer

hi < Paul Swithers> we ve gotten other reports of this issue and this seems to be a bug with running dbt with the ` log format json` flag which the dagster dbt library does to make it easier for us to parse out the log messages This is something that will need to be fixed from their end this bug is still present in the most recent version of dbt as far as I can tell We re working on a temporary workaround from our side that will hopefully make it into this week s release

owen · Answer

hey again < Paul Swithers> I ve dug into this a bit more and wasn t able to exactly reproduce the error that you were seeing although I got something similar What I was trying was creating two models `a sql` gt `select 1` and `b sql` gt `select from ref a ` Then I just ran `dbt log format json run select b` so b will select from a table that doesn t yet exist This gave me a JSON serialization error rather than a pickle error I ve opened up a PR on dbt s end to fix that issue but I just want to make sure that this PR will actually fix the issue you re seeing Is there anything you can think of that would be different about your setup that s leading you to hit a different error

Paul Swithers · Answer

I configure a dagster resource using the dbt cli resource from the dagster dbt package and run dbt commands using the cli function of the resulting dbt cli resource object

Paul Swithers · Answer

< owen> I had a chance to mimic your test and got the pickling error from the command line ```dbt log format json run select b``` ``` code Z028 data msg Pickling client objects is explicitly not supported invocation id b9a3c263 5888 43b6 b248 5177908ed7e6 level error log version 1 msg u001b 33mPickling client objects is explicitly not supported u001b 0m node info pid 93681 thread name MainThread ts 2022 06 30T17 54 36 030822Z type log line code Z029 data msg Clients have non trivial state that is local and unpickleable invocation id b9a3c263 5888 43b6 b248 5177908ed7e6 level error log version 1 msg Clients have non trivial state that is local and unpickleable node info pid 93681 thread name MainThread ts 2022 06 30T17 54 36 030988Z type log line ``` You were right There must be something about our set up that results in pickling exception messages where you get serialization messages Could it be that our project is built around BigQuery Either way the correct message should be this one 404 Not found Table cbh paul swithers dbt paul swithers a was not found in location US I got that when I ran this ```dbt run select b```