Davi
10/05/2022, 12:38 PMop
functions that mandatorily follow this input/output paradigm. Am I required to generate outputs and inputs in my functions to create a graph/job in Dagster? Currently I want to orchestrate functions that don't generate output.
Thank you all !Mat Brady
10/05/2022, 1:24 PMins
to another op
or use the functional dependency chaining of @job
definitions if it’s dependent e.g. op3(op2(op1()))
Example of what I mean by functional chaining
https://docs.dagster.io/tutorial/ops-jobs/connecting-ops#a-more-complex-dag
I’m using an @asset
pipeline and for that type there’s a specific `non_argument_deps`param to @asset
which tells it to depend on another @asset
without requiring that asset’s output if it generates any - as per what I’ve read @asset
s are built on top of @op
so I imagine this is the way `@op`s can be used. Example of non_argument_deps
https://docs.dagster.io/tutorial/assets/non-argument-deps#an-unzipped-csv-of-cereal-ratings
Hope this helps and is potentially correct :)Davi
10/05/2022, 3:30 PMMat Brady
10/05/2022, 3:41 PMfrom dagster import asset, OpExecutionContext
@asset
def asset1():
# do some stuff and don't return anything
pass
@asset(non_argument_deps={'asset1'}, required_resource_keys={'api'})
def asset2(context: OpExecutionContext):
# do some stuff and don't return anything
pass
@asset(non_argument_deps={'asset2'}, required_resource_keys={'api'})
def asset3(context: OpExecutionContext):
# do some stuff and don't return anything
pass
asset2
will run in parallel with asset1
?sandy
10/05/2022, 3:54 PMMat Brady
10/05/2022, 4:42 PMDavi
10/06/2022, 9:26 AM@op
def op_1():
# Do something
return Nothing
@op(ins={"depends": In(Nothing)})
def op_2():
# Do something
return Nothing
@op(ins={"depends": In(Nothing)})
def op_3():
# Do something
@job
def job():
dependency_link = op_1()
dependency_link = op_2(dependency_link)
op_3(dependency_link)