David Baur07/09/2021, 9:18 PM
Bryan Johnson07/10/2021, 1:55 AM
It’s quite hacky from a typing standpoint, but does work for skipping sub-graphs. There is probably a more correct way to do this.
if input is not None: # do normal logic yield Output(actual_output, "var_name") else: yield Output(None, "var_name")
alex07/12/2021, 3:49 PM
David Baur07/12/2021, 5:06 PM
alex07/12/2021, 5:22 PM
In the approaches you’re referring to that others have taken, are they typically generating a fixed set of dags well in advance of job submission and placing those files on their dagit instance, or is there a way to include the “pipeline DSL” text as part of the job submission?The most dynamic system I have seen used a database as the means of storing the “pipeline DSL”, then a fixed repository definition would fetch from the DB to create the working set of pipelines. In this example they authored the pipelines in a web gui that would persist to the database. I have not seen an example of more ephemeral pipelines. I can not think of any way to use the existing abstractions to stash the “pipeline DSL” in dagster managed places. You would need to manage the working set available to the fixed repository location that the dagster machinery is pointed at.
Bryan Johnson07/12/2021, 5:36 PM
alex07/12/2021, 5:57 PM
David Baur07/12/2021, 6:03 PM