https://dagster.io/ logo
#ask-community
Title
# ask-community
s

Simon

04/20/2022, 4:03 PM
https://docs.dagster.io/concepts/assets/asset-materializations#linking-assets-to-an-output-definition- sounds pretty good and interesting and I'd like to try it but then for a partitioned asset, how would this work? `Output`doesn't have a partition argument (https://docs.dagster.io/_apidocs/ops#dagster.Output)
Out
does (https://docs.dagster.io/_apidocs/ops#dagster.Out) but that seems like I'd be hardcoding it to a fixed partition (I don't really understand the usecase for this tbh). Is there something else I need to do to link an Op to a partitioned Asset?
j

johann

04/20/2022, 6:23 PM
cc @claire
c

claire

04/20/2022, 7:33 PM
Hi Simon! The easiest way to link an op to a partitioned asset is to generate an asset materialization with a partition key:
Copy code
@daily_partitioned_config(start_date=datetime(2020, 1, 1))
def my_partitioned_config(start: datetime, _end: datetime):
    return {
        "ops": {
            "my_op": {"config": {"date": start.strftime("%Y-%m-%d")}}
        }
    }

@op
def my_op(context):
    context.log_event(AssetMaterialization(asset_key=AssetKey("my_asset"), partition=context.op_config["date"]))

@job(config=my_partitioned_config)
def do_stuff_partitioned():
    my_op()
s

Simon

04/21/2022, 7:17 AM
@claire Thanks for the reply. Unfortunately from my testing that seems to result in something different compared to software defined assets. My guess would be that the old assets and software defined assets are different things? I'd like to make use of the software defined asset functionality for a partitioned asset using Ops. https://docs.dagster.io/concepts/assets/asset-materializations#linking-assets-to-an-output-definition- seems to suggest this should be possible, but then it only seems to work without partitions.
c

claire

04/21/2022, 4:21 PM
Hi Simon. If you want to create this functionality using software defined assets, you could do:
Copy code
daily_partitions_def = DailyPartitionsDefinition(start_date="2020-01-01")


@asset(partitions_def=daily_partitions_def)
def daily_partitioned_asset():
    return 1
Then if you execute the job for a certain partition, dagster will automatically generate an asset materialization for the asset partition.
s

Simon

04/21/2022, 4:23 PM
@claire That's correct, but unfortunately then it's no longer an
Op
and it seems like you can't combine
Op
with assets (or actually you can, see https://docs.dagster.io/concepts/assets/asset-materializations#linking-assets-to-an-output-definition- but not with partitions)
c

claire

04/21/2022, 4:26 PM
Yes. If you wanted to generate partitioned asset materializations within an op, the first code snippet I shared is probably the easiest way. That being said, we're currently working on a more generalized solution to link assets to op outputs, which should be released in the next couple months.
👍 1
s

Simon

04/21/2022, 4:32 PM
@claire That would be great! I expect in general bridging these two worlds (Ops vs software defined assets) would be a good improvement over the sort of two separate worlds they are now. If you want some input I'd be happy to provide our experiences/feedback 🙂 Along the same lines/a different solution to what we're trying to do is this https://github.com/dagster-io/dagster/issues/7089
c

claire

04/21/2022, 4:41 PM
Thanks Simon! The hope is to have an assets decorator that can decorate both graphs and ops.
👍 1
s

Simon

04/21/2022, 4:41 PM
That would be perfect 🙂
2 Views