Hi team, another question on SDA. Could a SDA depe...
# ask-community
Hi team, another question on SDA. Could a SDA depend on regular operations instead of assets?
or, can an asset job be broken down into smaller operations?
Hi Hebo. Yes, an SDA can depend on regular operations: • If you want multiple intermediate operations before outputting an asset, you could wrap these ops in a graph and create a graph-backed asset via AssetsDefinition.from_graph • If you want to define an op as an asset without refactoring the op to use
, you could use
Thanks Claire. I was using @asset(non_argument_deps=source_asset_key_strings) and define_asset_job() to convert it to a job. Could @graph takes in source asset keys as strings as well?
we are trying to link different jobs with assets. The jobs are dynamically generated so we don’t have a handle to the asset of upstream assets inside the graph/asset definition. I was trying to link thinks together by explicit asset keys
Hmm.. one thing you could try here is specifying an explicit input asset key via `AssetsDefinition.from_graph`:
Copy code
source_asset = SourceAsset(key="asset_one")

@op(ins={"upstream": In(Nothing)})
def asset_two():
    return 1

def my_graph(upstream):
    return asset_two(upstream)

graph_asset = AssetsDefinition.from_graph(
    my_graph, keys_by_input_name={"upstream": AssetKey("asset_one")}
If you want to define a non-argument dependency for a graph backed asset, you'll need to define a `Nothing` input on the op that has the upstream asset dependency.
Nice! For @graph decorator and the function my_graph, could I also pass in a dynamic list of upstreams? i tried this but it doesn’t work
Copy code
graph_ins = dict((key, In(Nothing)) for key in source_asset_keys)

        def my_graph():
This is the error
dagster._core.errors.DagsterInvalidDefinitionError: @graph 'my_graph' decorated function does not have parameter(s) 'source_asset_key1', which are in provided input_defs. @graph decorated functions should only have keyword arguments that match input names and, if system information is required, a first positional parameter named 'context'.
I don’t actually need to use the inputs, just need to build lineage
I saw GraphDefinition() but wondered if there is a way to get def my_graph() works
It unfortunately does not work, because the
decorated function must contain explicit argument names and pass them to ops within the function. This is because Dagster flattens graphs to build a flat mapping of ops, so any input to an op within the graph must be mapped to a graph input. You could try to programmatically define the graph: https://docs.dagster.io/concepts/ops-jobs-graphs/graphs#using-graphdefinitions
👍 1
Hi @claire, I am trying with GraphDefinition. In the graph, I included a dummy op “build_dependencies” as the only leaf node of the graph (by depending on all other nodes), which also takes in other assets as inputs. For the dependencies parameter, should I include the asset inputs for “build_dependencies” op? • If I don’t include them, I get and error from
“Detected conflicting node definitions with the same name”. This is due to the node_defs having “build_dependencies” with asset inputs and somehow another node_def was generated (I guess from dependencies parameter) which doesn’t have the asset inputs • If I include them, I get another error
Invalid dependencies: node "xxx" not found in node list
. I guess I’d need to somehow include source input assets to GraphDefinition?
Which dependencies parameter are you referring to? I think it might help for you to share the code with me if you don't mind, I'm not sure how your graph is being constructed