https://dagster.io/ logo
Title
s

Seth Kimmel

02/09/2023, 7:49 PM
Hey again! Is there any way to supply a dynamic io_manager_key to an asset? Specifically I want to switch between using a staging database in snowflake and a production one. I'm wrapping a group of assets in a
define_asset_job
object and passing a config, but it seems you are only able to pass config entries to the ops?
j

jamie

02/09/2023, 8:04 PM
would something like this work? may have some syntax errors (i wrote it in slack)
@asset(
   io_manager_key="snowflake_io_mgr"
)
def my_asset():
    ...

define_asset_job(name="staging_job", selection=[my_asset], resource_defs={"snowflake_io_mgr": snowflake_io_manager.configured(<staging_config>))

define_asset_job(name="prod_job", selection=[my_asset], resource_defs={"snowflake_io_mgr": snowflake_io_manager.configured(<prd_config>))
s

Seth Kimmel

02/09/2023, 8:39 PM
Hmm - just tried this, and it looks like
define_asset_job
doesn't have a
resource_defs
param
j

jamie

02/09/2023, 8:44 PM
ah yeah you’re totally right. i shouldnt try to write code from memory haha. can you try this instead?
@asset(
   io_manager_key="snowflake_io_mgr"
)
def my_asset():
    ...

define_asset_job(name="staging_job", selection=[my_asset], config={"resources": {"snowflake_io_mgr":{<staging_config}}})

define_asset_job(name="prod_job", selection=[my_asset], config={"resources": {"snowflake_io_mgr":{<staging_config}}})
if your repository/definitions when you set up the mapping of resource to resource key, you won’t use the
.configured
api, instead all the configuration will be provided as the
config
arg in
define_asset_job
s

Seth Kimmel

02/09/2023, 9:01 PM
Something is still slightly off - but it feels close! It's saying that my
fs_io_manager
is required by the first op, but not provided. What would make it recognize the resource in the
Definitions
class, but not in the config?
j

jamie

02/09/2023, 9:02 PM
can you share your
define_asset_job
and your
Definitions
?
s

Seth Kimmel

02/09/2023, 9:05 PM
I'll shoot you a DM!
:ty-spinny: 1