greetings to all. Working with

dagster-dbt

and SDAs and I bumped into an issue. When I attempt to materialize multiple assets and their corresponding dbt run ends at approx the same time, the returned output is correct for only one of the assets and incorrect for the rest. The reason is that the returned output is read from the same file

run_results.json

. In my partiicular case all these assets are materialized as views in dbt so this issue is very likely to occur. I guess one way to contain this is using

tag_concurrency_limits

, other/better suggestions are welcome 😄

hey @Dimitris Stafylarakis! that makes sense -- are these completely separate asset jobs, or just different runs of the same asset jobs that are getting kicked off at the same time?

hi @owen!

that makes sense -- are these completely separate asset jobs, or just different runs of the same asset jobs that are getting kicked off at the same time?

the latter, I’m just selecting a bunch of assets and materializing them simultaneously. I’d say that using the dbt cli for concurrent runs feels rather shaky; any two or more dbt jobs that might finish at the same time will face the same problem. Since I’m running dagster on k8s I’ll try using the k8s_job_executor, this should offer better isolation.

another potential solution here would be if the dagster-dbt library copied the dbt project into a temporary directory before executing anything 🤔 . would definitely be more efficient than forcing people to isolate execution to entirely different pods. I think you could achieve that behavior by subclassing the DbtCLIResource if you were so inclined, but this is potentially something that should just be built in to the library itself

or maybe prefix the target dir per run?

that was what I wanted to be able to do (seems so natural!) but it doesn't seem like dbt lets you configure that through a CLI flag -- you can only set the target dir at the project level

hm, this issue mentions that we can use vars passed from the cli to set target-path. Didn’t know that, I’ll give it a try

ah interesting -- let me know if that works

it does 👍 no need for subclassing then 🙂

awesome

hm I spoke too soon @owen. while dbt itself handles variable target paths just fine, dagster uses the target_path configured with the dbt resource. Since we can’t override this per op, isolated dbt runs are unfortunately not yet possible.