Hello is there a canonical way to attach a piece of config o dagster #ask-community

Hello, is there a canonical way to attach a piece ...

Arvind Narayan

08/25/2023, 3:35 PM

Hello, is there a canonical way to attach a piece of config or metadata to an asset such that a downstream assets op is able to access the config of its dependency asset? For instance…

Copy code

@asset
def hackernews_top_story_ids(config: FileAssetConfig)-> str:

@asset
def hackernews_top_stories(context: AssetExecutionContext, 
                           hackernews_top_story_ids: str,
                           config: MultiFrameAssetConfig,
                           ) -> pd.DataFrame:

materialize(
    assets = [hackernews_top_story_ids, hackernews_top_stories],
    resources = {
        'io_manager': datastore_io_manager()
    },
    run_config = RunConfig(
        ops = {
            'hackernews_top_story_ids': FileAssetConfig(name="hackernews_top_story_ids",
                                                        workspace="dagster-test"),
            'hackernews_top_stories': MultiFrameAssetConfig(name="hackernews_top_stories",
                                                          workspace="dagster-test",
                                                          concat_dim="story_id"),
        }
    )
)

In the materialization of the first step, I am able to access the FileAssetConfig object as

context.step_context.op_config

. How can I get the same config in the second step?

context.upstream_output...???

Arvind Narayan

08/25/2023, 4:21 PM

To elaborate… I would like the Asset definition to maintain some metadata that maps the asset object in dagster to an artifact in some other system. The artifact may have a different name in the other system. I was hoping that attaching some configuration to the asset would enable me to determine what its mapped entity in the other system is and to be able to load it.

jamie

08/25/2023, 4:28 PM

hey @Arvind Narayan there isn’t a super great way to do this kind of thing at the moment, but you can nest config within config so you could do something like this

Copy code

class SharedConfig(Config):
    foo: str
    bar: str 

class FirstConfig(Config):
     shared_config: SharedConfig 
     baz: str

class SecondConfig(Config):
     shared_config: SharedConfig 
     qux: str

shared_config = SharedConfig(foo="hello", bar="world")

materialize(
   assets=[...],
   run_config=RunConfig(ops={"first_op": FirstConfig(shared_config=shared_config, baz="!")
)

Arvind Narayan

08/28/2023, 4:28 PM

Thank you @jamie, let me give that a shot.

Open in Slack

Previous Next