Jeremy
07/18/2022, 8:50 PMjobby_job_prod = AssetsDefinition.from_graph(
jobby_job_graph,
partitions_def=HourlyPartitionsDefinition(start_date=PARTITION_START_DATE),
keys_by_output_name={"result": AssetKey(["augmentation_data", "i_metadata"])},
)
how do i define non_argument_deps for AssetsDefinition.from_graph ? (edited)
there’s an upstream dep that’s not an argument passed in non_argument_deps
. Isn’t an option in AssetsDefinition.from_graph
owen
07/18/2022, 9:29 PMfrom_graph()
call at the moment. Under the hood, the @asset
decorator is creating a new Op, with Nothing-type inputs corresponding to each of the non_argument_deps
. Because the from_graph
method works on existing graphs (rather than creating new ones), it doesn't do this step.
To work around it, one solution would be to add a new Nothing input to the ip_augmentation_graph and wire it to the first op in the graph, i.e.:
# before ***************
@op
def first_op(x):
...
@graph
def graph_asset(regular_asset):
return second_op(first_op(regular_asset))
# after ****************
@op(ins={"x": In(), "start_after": In(Nothing)})
def first_op(x):
...
@graph
def graph_asset(reg_asset, non_arg_asset):
return second_op(first_op(regular_asset, start_after=non_arg_asset))