https://dagster.io/ logo
Title
r

Rafael Gomes

12/10/2022, 6:54 PM
I always get confused about when should I use
Partition
Assets or
Partition
Jobs. What are the best practices ?
f

Fraser Marlow

12/10/2022, 11:22 PM
Is your question “When should I use Assets vs Jobs”? Or was there something specific to partitioning?
r

Rafael Gomes

12/10/2022, 11:41 PM
It is related to partitioning. For example, I can have a multi-partitioned asset that downloads a file daily from external resources. I can achieve the same result by having a nonpartitioned asset but a partitioned job that will trigger the same asset several times (just changing the config_schema). Maybe I'm overcomplicating things here
s

sandy

12/12/2022, 10:25 PM
We generally recommend using assets whenever your goal is to produce a file/table/ML model/other data asset It's common to use partitioned assets with partitioned jobs: i.e. your asset has a partition for each day, and the job materializes the partition for that day when it runs Here's documentation on how to do this: https://docs.dagster.io/concepts/partitions-schedules-sensors/partitions#partitioned-asset-jobs You could then use
build_schedule_from_partitioned_job
to create a schedule that runs daily: https://docs.dagster.io/_apidocs/schedules-sensors#dagster.build_schedule_from_partitioned_job
❤️ 1