https://dagster.io/ logo
b

Basil V

04/02/2020, 12:03 AM
Hi, I'm running into an issue in which I've created a dependency structure between a few composite_solids by passing Nothing between them (since I don't have real values to pass between them), however, I expected the dependency structure to be enforced for their child solids as well, but this doesn't seem to be the case. For instance, I have two composite_solids. I expect all the child solids within my first composite_solid to finish before any of the child solids in my second composite_solid begin. However, the dependency structure is only being enforced for the child solid who's return value is returned by the composite. The other child solids in my second composite_solid start at the same time as the child solids in my first composite_solid. Please let me know if what I've described makes sense sorry if that is a bit verbose/confusing. Thanks so much for any help!
m

max

04/02/2020, 12:07 AM
@alex I think we have an open issue for this
a

alex

04/02/2020, 12:07 AM
ya - you have to treat you
Nothing
dependency like it was real data and flow it all the way to all the solids you want to block
b

Basil V

04/02/2020, 12:08 AM
Oh I have to pass it to all the child solids as well?
a

alex

04/02/2020, 12:09 AM
ya - I think the right mental model is treat as a piece of data that solids need to have before they can execute
given what you said - you at least need to flow it to all the origin nodes within a composite
maybe that means all the children but depends on the shape of the dags in the composites
and to @max’s point we have an issue tracking that this whole
Nothing
business is quite cumbersome and confusing
b

Basil V

04/02/2020, 12:11 AM
Ok that makes sense, good to know. Thanks for the help. Do you know the issue # so I can follow. This will work for a short term solution but yeah the Nothing stuff w/ composites doesn't feel ideal. Would love to see something a bit cleaner later on, and happy to help contribute if I can.
b

Basil V

04/02/2020, 12:12 AM
Awesome thanks!
a

alex

04/02/2020, 12:14 AM
ya it would be great if you could add a comment on that issue with a simplified version of what you tried to do and why you found it unpleasant
b

Basil V

04/02/2020, 12:14 AM
Definitely will do
a

alex

04/02/2020, 12:14 AM
just so we have more data points to factor in to our eventual solution
👍 1
b

Basil V

04/02/2020, 2:17 AM
May actually have something blocking me from flowing my
Nothing
dependency to my child solid in a specific case—I'm using dagster-dbt and I can't change the input defs to map
Nothing
for the dbt_solid I'm using within my composite solid (I left a comment on that issue with much more detail if anything is unclear). Let me know if you have any suggested workarounds to this. Thanks again!
a
b

Basil V

04/02/2020, 5:12 PM
Thanks for that! I'm not actually familiar with Phabricator—is this a change I should just look for in the next release? In the meantime, I'll just run a local copy of Dagster and make this change myself. Thanks again.
a

alex

04/02/2020, 5:19 PM
phabricator is just what we use for code review - it will land to github shortly and likely make it out in a release today
b

Basil V

04/02/2020, 5:20 PM
Awesome. Thanks again