https://dagster.io/ logo
#ask-community
Title
# ask-community
c

Chaitya

04/26/2023, 1:22 AM
Hey folks, I'm trying to include multiple resources with my call to
materialize
but can't seem to get it working with the new pythonic API. Details in thread.
I've got an IO Manager and a normal resource.
Copy code
class MyIOManagerResource(ConfigurableResourceFactory[MyIOManager]):

    base_path: str

    def create_resource(self, context: InitResourceContext) -> MyIOManager:
        return create_my_io_manager(self.base_path)
Copy code
class MyResource(ConfigurableResource["MyResource"]):
    some: str
    fields: str
In my call to materialize previously, I would construct a config dictionary and call materialize like this:
Copy code
run_config = {
    "resources": {
        "io_manager": {
            "config": {
                "base_path": "/tmp"
            }
        },
        "my_resource": {
            "config": {
                 "some": "value1",
                 "fields": "value2",
            }
        }
    }
}

_ = materialize([my_asset], run_config=run_config, resources={ "io_manager": create_my_io_manager, "my_resource": MyResource.configure_at_launch()})
This worked as I expected. Now trying to switch to the new pythonic configuration:
Copy code
_ = materialize([my_asset], run_config=RunConfig(resources={"io_manager": MyIOManagerResource(base_path="/tmp"), "my_resource": MyResource(some="value1", fields="value2")}), resources={ "io_manager": create_my_io_manager, "my_resource": MyResource.configure_at_launch()})
This fails with:
Copy code
Error 1: Received unexpected config entry "my_resource" at path root:resources. Expected: "{ io_manager?: { config?: Any } }".
If I remove the
my_resource
key in the RunConfig constructor:
Copy code
_ = materialize([my_asset], run_config=RunConfig(resources={"io_manager": MyIOManagerResource(base_path="/tmp")}), resources={ "io_manager": create_my_io_manager, "my_resource": MyResource.configure_at_launch()})
this runs successfully. But unfortunately, I cannot run this without
my_resource
. Is this a bug? How can i pass multiple resources to the materialize call? I'm not sure I need the
MyResource.configure_at_launch()
anymore, but i'm not really sure if thats causing my issue.
headbang It looks like it was an issue in my materialize call. My list of assets was actually empty in reality. Really weird that the error I got was a schema error for the config instead of something telling me "you're not actually running anything and you're supplying an extra input that isnt used"
Adding an asset to my list resolved the issue.
c

chris

04/26/2023, 5:39 PM
Yea I think you just hit a weird corner case with materialize unfortunately. Filed an issue
🙏 1
2 Views