https://dagster.io/ logo
#ask-community
Title
# ask-community
h

Hebo Yang

08/09/2022, 6:27 PM
Hi team, another question on SDA. Could a SDA depend on regular operations instead of assets?
or, can an asset job be broken down into smaller operations?
c

claire

08/09/2022, 6:48 PM
Hi Hebo. Yes, an SDA can depend on regular operations: • If you want multiple intermediate operations before outputting an asset, you could wrap these ops in a graph and create a graph-backed asset via AssetsDefinition.from_graph • If you want to define an op as an asset without refactoring the op to use
@asset
, you could use
AssetsDefinition.from_op
h

Hebo Yang

08/09/2022, 7:12 PM
Thanks Claire. I was using @asset(non_argument_deps=source_asset_key_strings) and define_asset_job() to convert it to a job. Could @graph takes in source asset keys as strings as well?
we are trying to link different jobs with assets. The jobs are dynamically generated so we don’t have a handle to the asset of upstream assets inside the graph/asset definition. I was trying to link thinks together by explicit asset keys
c

claire

08/09/2022, 7:28 PM
Hmm.. one thing you could try here is specifying an explicit input asset key via `AssetsDefinition.from_graph`:
Copy code
source_asset = SourceAsset(key="asset_one")


@op(ins={"upstream": In(Nothing)})
def asset_two():
    return 1


@graph
def my_graph(upstream):
    return asset_two(upstream)


graph_asset = AssetsDefinition.from_graph(
    my_graph, keys_by_input_name={"upstream": AssetKey("asset_one")}
)
If you want to define a non-argument dependency for a graph backed asset, you'll need to define a `Nothing` input on the op that has the upstream asset dependency.
h

Hebo Yang

08/09/2022, 8:02 PM
Nice! For @graph decorator and the function my_graph, could I also pass in a dynamic list of upstreams? i tried this but it doesn’t work
Copy code
graph_ins = dict((key, In(Nothing)) for key in source_asset_keys)

        @graph(
            ins=graph_ins
        )
        def my_graph():
This is the error
dagster._core.errors.DagsterInvalidDefinitionError: @graph 'my_graph' decorated function does not have parameter(s) 'source_asset_key1', which are in provided input_defs. @graph decorated functions should only have keyword arguments that match input names and, if system information is required, a first positional parameter named 'context'.
I don’t actually need to use the inputs, just need to build lineage
I saw GraphDefinition() but wondered if there is a way to get def my_graph() works
c

claire

08/09/2022, 8:40 PM
It unfortunately does not work, because the
@graph
decorated function must contain explicit argument names and pass them to ops within the function. This is because Dagster flattens graphs to build a flat mapping of ops, so any input to an op within the graph must be mapped to a graph input. You could try to programmatically define the graph: https://docs.dagster.io/concepts/ops-jobs-graphs/graphs#using-graphdefinitions
👍 1
h

Hebo Yang

08/12/2022, 10:27 PM
Hi @claire, I am trying with GraphDefinition. In the graph, I included a dummy op “build_dependencies” as the only leaf node of the graph (by depending on all other nodes), which also takes in other assets as inputs. For the dependencies parameter, should I include the asset inputs for “build_dependencies” op? • If I don’t include them, I get and error from
_build_all_node_defs
“Detected conflicting node definitions with the same name”. This is due to the node_defs having “build_dependencies” with asset inputs and somehow another node_def was generated (I guess from dependencies parameter) which doesn’t have the asset inputs • If I include them, I get another error
Invalid dependencies: node "xxx" not found in node list
. I guess I’d need to somehow include source input assets to GraphDefinition?
c

claire

08/12/2022, 10:32 PM
Which dependencies parameter are you referring to? I think it might help for you to share the code with me if you don't mind, I'm not sure how your graph is being constructed
2 Views