Hi, I am new to Dagster. Is there any way to confi...
# announcements
Hi, I am new to Dagster. Is there any way to configure a pipeline so that a certain solid within it is executed/not executed based on a boolean?
Hey Rubén, this example might be helpful: https://docs.dagster.io/examples/conditional_execution
Thanks for the swift response! However, I know beforehand if I want to execute a solid or not. To provide some context, I have two solids, one that uploads data to Drive (using pydrive) and another one that just stores the data locally (so I do not need pydrive). I want to be able to tell the pipeline whether to store data in Drive or locally before running the pipeline
Maybe there is an alternative approach (having 2 different pipelines does not seem the way to go for me, but maybe it is)
a lot of apis take a
argument which could be used to selectively execute
👍 1
Ah I see, yeah there are a lot of ways you could approach this. You could use solid selection as Alex said (https://docs.dagster.io/overview/solid-selection), or you could add a “should_execute” boolean field in the config schema for both solids if you want to use pipeline config to control this
Also, if you’re varying whether to upload to drive or download locally based on environment (dev/prod), you may also want to take a look at modes and resources https://docs.dagster.io/overview/modes-resources-presets/modes-resources
Wow, thanks guys, I will have a look at solid selection! The second approach seems to delegate the execution logic to the solids themselves, since you would have to handle the should_execute param once the solid is being executed:
Copy code
if should_execute == False:
We have this at the moment, but feels weird
We were debating on using mode def vs another alternative, so based on your help we will decide one of them 🙂
Thanks a lot
That’s correct, and you’re right that is weird
Let us know if you have any more questions!