hello, i noticed something odd this morning - I st...
# dagster-serverless
l
hello, i noticed something odd this morning - I started two runs that materialize separate dbt assets from the same project at around the same time (this was manually, so there was a few seconds difference between them). One of them went off without a hitch (lets call this run X) but the other one (run Y) yielded warnings of the form
Materializing unexpected asset key: AssetKey([a,b])
where
a,b
is an asset from run X! Also, while asset materialization events for run X were successfully logged, there were no such events for run Y even though the logs indicate that the dbt job was successful. I notice that the run has
isolation:disabled
so i suspect this has something to do with it - any advice on what happened and how to prevent it from happening in the future?
j
hey @Leo Qin can you reproduce this error? Does running with
isolation:enabled
fix it?
l
trying it again without isolation... can you direct me to
j
https://docs.dagster.io/dagster-cloud/deployment/serverless#run-isolation depending on how you're launching the run you can either toggle the "isolate run environment" checkbox or add
Copy code
non_isolated_runs:
  enabled: True
to the deployment settings
@Leo Qin you can also set
"dagster/isolation", "disabled"
as a tag on the run if you keep the deployment setting enabled
l
ok, so the retry without isolation actually failed... the failed run
unexpectedly exited with exit code -9
j
hmmmm is there a stack trace you can share?
l
yeah, dm them?
j
sure!
l
@Joe - starting the two runs with isolation lead to successful runs, and the outputs look correct as well.
🎉 1
j
nice Ill followup with why the non-isolated fails
@Leo Qin I got an answer for why the non-isolated runs fail: dbt will write the output that dagster parses to the same filepath every time, for non-isolated runs that are started at the same time this creates a race condition where whichever job run completes second will overwrite the first ones output, causing that failure.
l
ah - thanks for the followup. that makes sense!