Hello curious if anyone has seen this before I m using Dagst dagster #ask-community

Hello - curious if anyone has seen this before: I...

James Hale

02/02/2023, 2:40 AM

Hello - curious if anyone has seen this before: I'm using Dagster with dbt on Dagster Serverless connected to Snowflake. When running dbt, the queries resolve in <1s, but are consistently taking ~30s to resolve in

dbt build

. This doesn't occur when running dbt locally. E.g. in the image attached.

jamie

02/02/2023, 3:39 PM

cc @rex

James Hale

02/06/2023, 6:10 PM

@jamie @rex any thoughts here?

rex

02/06/2023, 7:00 PM

Could you run your dbt commands in Serverless with

--debug

enabled? Our serverless tasks should have more than enough CPU/MEM to handle dbt tasks, esp since the transformation is happening in the data warehouse anyways.

rex

02/06/2023, 7:03 PM

You could also try turning off usage stats from being sent on `dbt`’s side to see if that alleviates some latency: https://docs.getdbt.com/reference/global-configs#send-anonymous-usage-stats

rex

02/06/2023, 7:06 PM

also, dbt uses some caching to help out with running models in local development, and that cache might not be hit when run in an isolated environment (like serverless): https://docs.getdbt.com/reference/global-configs#cache-database-objects-for-selected-resource

James Hale

03/02/2023, 2:24 PM

@rex looping back around on this - I'll take a look at both above. However, I've been able to correlate the change with moving from

dagster-cloud=1.0.13

dagster-cloud=1.1.20

. Was there, perhaps, a change to the dockerfile for serverless deploys in that interval?

James Hale

03/03/2023, 3:29 PM

@rex @jamie could you advise on how to access dbt's

--debug

flag in the

dagster-dbt

integration? It doesn't appear to be exposed as an option: https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/cli/constants.py

2 Views

Open in Slack

Previous Next