https://dagster.io/ logo
o

Oleg Agapov

12/17/2020, 5:42 PM
Hi guys, One thing I don't understand about Dagster. What should I do if I need to make dependent solids if they don't have outputs? Similar in Airflow is
Copy code
task1 >> task2 >> task3
My solids are kinda stateless (don't return any values), but their order is important.
s

sandy

12/17/2020, 5:43 PM
Hey Oleg, here's an example: https://docs.dagster.io/examples/nothing Let me know if you have further questions
d

dwall

12/17/2020, 5:45 PM
on the topic of
Nothing
- one thing I thought about the other day is that this will probably be a pretty common pattern in the long term (using
Nothing
to model task ordering dependencies between solids), so do you think it's worth establishing best practices around nomeclature for these types of inputs and outputs?
o

Oleg Agapov

12/17/2020, 5:45 PM
Oh, cool. Thanks Sandy
@sandy is there any sugar for piping? similar to pandas
Copy code
df.pipe(transform1).pipe(transform2)
? so to have
Copy code
solid1.pipe(solid2).pipe(solid3)
if my previous question is not clear I have 10 tasks, without outputs, but have to be triggered one by one currently I have to do
Copy code
solid10(solid9(...solid1()))
I'm trying to make it more visual and usable
m

max

12/17/2020, 7:17 PM
^ there isn't currently sugar for this
we have previously considered stuff like
solid10.after(solid9)
etc, but obviously that doesn't reverse the order like
pipe
m

mrdavidlaing

12/21/2020, 3:23 PM
@Oleg Agapov - I find myself with a similar desire for some kind of pipe() sugar to chain sets of tasks together. However; I wonder if this is actually a something of a dagster code-smell; since it the means that the solids in the resulting pipeline will be harder to run independently (for unit testing & re-running sections of the pipeline with “intermediate data”)
@max Was it concerns like this that prevented the implementation of things like solid.after() ?
m

max

12/21/2020, 6:56 PM
well, yes, if you can model the dependencies between solids using data you should 🙂
👍 1
sometimes sentinel values or pointer values are appropriate