What's best pattern for creating reusable ops/grap...
# ask-community
t
What's best pattern for creating reusable ops/graphs with generic resource keys?
resource_defs
is an argument to
AssetsDefinition.form_graph()
but not for
AssetsDefinition.from_op()
which prevents reusing the same op since you cannot pass specific resources in when creating the asset. We have multiple configured snowflake resources which is problematic when trying to create asset definitions using an op.
I've been wrapping the op within a graph as a workaround
Copy code
@op(required_resource_keys={"snowflake_client"})
def example_op():
    ...

@graph
def example_graph():
    example_op()

# cannot pass in specific snowflake client
asset_from_op = AssetsDefinition.from_op(example_op)

# can pass any of the configured snowflake resources in as the 'snowflake_client'
asset1_from_graph = AssetsDefinition.from_graph(
    example_graph,
    resource_defs={"snowflake_client": get_resource("specific_snowflake_resource_A")},
)

asset2_from_graph = AssetsDefinition.from_graph(
    example_graph,
    resource_defs={"snowflake_client": get_resource("specific_snowflake_resource_B")},
)
in this example,
get_resource
is an internal tool which takes the key passed in and, based on the environment, gets the resource from a dictionary similar to dagster's example repo
o
Hi @Tom Reilly! I don't see a particular reason that
from_graph
would have a
resource_defs
argument, but not
from_op
. This was likely an oversight, but it would be a fairly straightforward change if you're interested in contributing. Otherwise, you should be able to do
Copy code
asset_from_op = AssetsDefinition.from_op(example_op).with_resources({"snowflake_client": ...})
as a workaround
thank you box 1