https://dagster.io/ logo
Title
g

geoHeil

02/09/2022, 9:06 AM
With the upcoming 0.14 edition of dagster will partitioned (software-defined) assets be part of the API?
y

yuhan

02/09/2022, 5:03 PM
cc @sandy
s

sandy

02/09/2022, 5:04 PM
yup! in fact, partitioned assets are available in preview in our most recent release
g

geoHeil

02/09/2022, 5:06 PM
I know. But so far I did not figure out how to use partitions in software defined assets. Would you have an example for me?
s

sandy

02/09/2022, 5:09 PM
it's not the most concise example, but the hacker news assets demo contains partitioned assets. here's a definition for one: https://github.com/dagster-io/dagster/blob/master/examples/hacker_news_assets/hacker_news_assets/assets/items.py and here's a job that includes some: https://github.com/dagster-io/dagster/blob/master/examples/hacker_news_assets/hacker_news_assets/jobs/hacker_news_api_download.py
I'd be happy to walk through it on a zoom if it would be helpful
g

geoHeil

02/09/2022, 5:17 PM
Thanks - I think the two links are great examples to what I need.
However, how is the mapping of partitions performed from one asset to the next one (downstream in the lineages). Is it a must, that they are all partitioned in the same way?
s

sandy

02/09/2022, 5:29 PM
currently, all the assets within the same job must have the same partitioning (or no partitioning). we have a more flexible notion of a `PartitionMapping`: https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/core/asset_defs/partition_mapping.py, an important subclass of which is `TimeWindowPartitionMapping`: https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/core/asset_defs/time_window_partition_mapping.py
however partition mappings currently only work across jobs
g

geoHeil

02/09/2022, 6:27 PM
Do you have any plans to support a mix of non-partitioned and partitioned assets?
I.e. a daily/hourly job with enrichment of some non-partitioned metadata
s

sandy

02/09/2022, 6:28 PM
with our current apis, you can combine partitioned and non-partitioned assets in a single job
😛artydagster: 1