https://dagster.io/ logo
Title
a

André Augusto

11/21/2022, 9:31 PM
hey! it is possible to have asset partitions defined at only the
job
level? I don’t want to define them at the asset-level, since it is easier (and more DRY) to define them at only the job level. The following code (almost the same as the Parttioned Asset Jobs page) yields no partitions in dagit (v.1.1.2)
from dagster import AssetSelection, HourlyPartitionsDefinition, asset, define_asset_job

hourly_partitions_def = HourlyPartitionsDefinition(start_date="2022-05-31-00:00")


@asset
def asset1():
    ...


@asset
def asset2():
    ...


partitioned_asset_job = define_asset_job(
    name="asset_1_and_2_job",
    selection=AssetSelection.assets(asset1, asset2),
    partitions_def=hourly_partitions_def,
)
s

sandy

11/21/2022, 10:27 PM
Hey Andre - this isn't currently possible. You'd want the same assets to have different partitions in different jobs?
n

nickvazz

11/22/2022, 12:30 AM
I think my question/feature-request would be handled with job partitions as well https://dagster.slack.com/archives/C01U954MEER/p1669061141407199
a

André Augusto

11/22/2022, 11:09 AM
Hey Sandy! its kind of confusing (and maybe dangerous), but yes. Let me give you a short example. I have a job that is somewhat like this:
A -> B -> D
     C /
and another one that is like this:
A -> B  -- > Z
X -> Y /
(all those letters are
dbt
assets. In the second job
Z
depends on both
B
and
Y
). In this scenario, only the
D
and the
Z
assets are user-facing. By our users requirements, they don’t mind the
D
asset being refreshed once a day. However, for the
Z
asset in the second job, they want it to be hourly. Today, I can do it “manually” by creating schedules and triggering the jobs at the desired frequency. However, I wanted to use partitions, since it gives me greater control of what ran and what not (hence the question that I asked). Lastly, this feature request may sound like an anti-pattern (or thing to be avoided), but all my assets are idempotent. And you can interpret this
job_partition
as the partition for our final assets (in this case,
D
and
Z
), since are those that my users care about (and not the previous ones; think something like the Medallion architecure).