Hi team, another question on SDA. Could a SDA depend on regular operations instead of assets?

Hi Hebo. Yes, an SDA can depend on regular operations: • If you want multiple intermediate operations before outputting an asset, you could wrap these ops in a graph and create a graph-backed asset via AssetsDefinition.from_graph • If you want to define an op as an asset without refactoring the op to use

@asset

, you could use

AssetsDefinition.from_op

Thanks Claire. I was using @asset(non_argument_deps=source_asset_key_strings) and define_asset_job() to convert it to a job. Could @graph takes in source asset keys as strings as well?

Hmm.. one thing you could try here is specifying an explicit input asset key via `AssetsDefinition.from_graph`:

source_asset = SourceAsset(key="asset_one")


@op(ins={"upstream": In(Nothing)})
def asset_two():
    return 1


@graph
def my_graph(upstream):
    return asset_two(upstream)


graph_asset = AssetsDefinition.from_graph(
    my_graph, keys_by_input_name={"upstream": AssetKey("asset_one")}
)

If you want to define a non-argument dependency for a graph backed asset, you'll need to define a `Nothing` input on the op that has the upstream asset dependency.

Nice! For @graph decorator and the function my_graph, could I also pass in a dynamic list of upstreams? i tried this but it doesn’t work

graph_ins = dict((key, In(Nothing)) for key in source_asset_keys)

        @graph(
            ins=graph_ins
        )
        def my_graph():

This is the error

dagster._core.errors.DagsterInvalidDefinitionError: @graph 'my_graph' decorated function does not have parameter(s) 'source_asset_key1', which are in provided input_defs. @graph decorated functions should only have keyword arguments that match input names and, if system information is required, a first positional parameter named 'context'.

It unfortunately does not work, because the

@graph

decorated function must contain explicit argument names and pass them to ops within the function. This is because Dagster flattens graphs to build a flat mapping of ops, so any input to an op within the graph must be mapped to a graph input. You could try to programmatically define the graph: https://docs.dagster.io/concepts/ops-jobs-graphs/graphs#using-graphdefinitions

Hi @claire, I am trying with GraphDefinition. In the graph, I included a dummy op “build_dependencies” as the only leaf node of the graph (by depending on all other nodes), which also takes in other assets as inputs. For the dependencies parameter, should I include the asset inputs for “build_dependencies” op? • If I don’t include them, I get and error from

_build_all_node_defs

“Detected conflicting node definitions with the same name”. This is due to the node_defs having “build_dependencies” with asset inputs and somehow another node_def was generated (I guess from dependencies parameter) which doesn’t have the asset inputs • If I include them, I get another error

Invalid dependencies: node "xxx" not found in node list

. I guess I’d need to somehow include source input assets to GraphDefinition?

Which dependencies parameter are you referring to? I think it might help for you to share the code with me if you don't mind, I'm not sure how your graph is being constructed