Chris Histe
11/16/2022, 4:09 PMgcs_pickle_io_manager
IO Manager. Here’s a close version of my code:
import os
from dagster import op, graph, AssetsDefinition, ResourceDefinition
from dagster_gcp import bigquery_resource, gcs_resource
from dagster_gcp.gcs.io_manager import gcs_pickle_io_manager
@op
def upstream_op():
return 1
@op
def downstream_op(upstream_op):
return upstream_op + 1
@graph
def my_graph():
return downstream_op(upstream_op())
asset_definition = AssetsDefinition.from_graph(
my_graph,
group_name="my_group",
resource_defs={
"gcs_bucket": ResourceDefinition.hardcoded_resource(os.getenv("BUCKET", "bucket")),
"gcs": gcs_resource,
"bigquery": bigquery_resource,
"io_manager": gcs_pickle_io_manager,
},
)
The error I’m getting:
Conflicting versions of resource with key 'io_manager' were provided to different assets. When constructing a job, all resource definitions provided to assets must match by reference equality for a given key.
I’m not sure I understand what this error means.
I tried a couple different things but I was not able to fix it 😞
Any help is greatly appreciatedclaire
11/16/2022, 8:16 PMwith_resources
?
This issue often arises because resources are scoped at the repository level (in order to power the global "materialize" button) so a given resource key must have the same value throughout the repository.
One workaround would be to provide a per-output IO manager key for your graph-backed asset, like the example here.Chris Histe
11/17/2022, 4:38 PMload_assets_from_package_module
would detect AssetsDefinition.from_graph
. I managed to fix the bug and make our code way simpler and cleaner 🙂 thanks