https://dagster.io/ logo
Title
c

Chaitya

04/26/2023, 1:22 AM
Hey folks, I'm trying to include multiple resources with my call to
materialize
but can't seem to get it working with the new pythonic API. Details in thread.
I've got an IO Manager and a normal resource.
class MyIOManagerResource(ConfigurableResourceFactory[MyIOManager]):

    base_path: str

    def create_resource(self, context: InitResourceContext) -> MyIOManager:
        return create_my_io_manager(self.base_path)
class MyResource(ConfigurableResource["MyResource"]):
    some: str
    fields: str
In my call to materialize previously, I would construct a config dictionary and call materialize like this:
run_config = {
    "resources": {
        "io_manager": {
            "config": {
                "base_path": "/tmp"
            }
        },
        "my_resource": {
            "config": {
                 "some": "value1",
                 "fields": "value2",
            }
        }
    }
}

_ = materialize([my_asset], run_config=run_config, resources={ "io_manager": create_my_io_manager, "my_resource": MyResource.configure_at_launch()})
This worked as I expected. Now trying to switch to the new pythonic configuration:
_ = materialize([my_asset], run_config=RunConfig(resources={"io_manager": MyIOManagerResource(base_path="/tmp"), "my_resource": MyResource(some="value1", fields="value2")}), resources={ "io_manager": create_my_io_manager, "my_resource": MyResource.configure_at_launch()})
This fails with:
Error 1: Received unexpected config entry "my_resource" at path root:resources. Expected: "{ io_manager?: { config?: Any } }".
If I remove the
my_resource
key in the RunConfig constructor:
_ = materialize([my_asset], run_config=RunConfig(resources={"io_manager": MyIOManagerResource(base_path="/tmp")}), resources={ "io_manager": create_my_io_manager, "my_resource": MyResource.configure_at_launch()})
this runs successfully. But unfortunately, I cannot run this without
my_resource
. Is this a bug? How can i pass multiple resources to the materialize call? I'm not sure I need the
MyResource.configure_at_launch()
anymore, but i'm not really sure if thats causing my issue.
:headbang: It looks like it was an issue in my materialize call. My list of assets was actually empty in reality. Really weird that the error I got was a schema error for the config instead of something telling me "you're not actually running anything and you're supplying an extra input that isnt used"
Adding an asset to my list resolved the issue.
c

chris

04/26/2023, 5:39 PM
Yea I think you just hit a weird corner case with materialize unfortunately. Filed an issue
🙏 1