If I have a Job that runs many ops, I want to part...
# ask-community
If I have a Job that runs many ops, I want to partition that job on a daily basis, and then have another op that would compare yesterdays events to today's events. I was looking here - https://docs.dagster.io/concepts/partitions-schedules-sensors/partitions#date-partitioned-job and here https://docs.dagster.io/concepts/partitions-schedules-sensors/partitions#partitionmappings, but I am a little confused on how to tie it all together, maybe even as 1 single job, like
I have exactly the same question. I want to compute diffs between the last two days’ data.
i think you may run today’s partition, and in the end have an op to pull today’s result and yday’s result based on the partition_key.
here’s a similar answer: https://github.com/dagster-io/dagster/discussions/10264#discussioncomment-4038325 which has
, instead, you can
, and fetch some metadata from there.
I think you could achieve this with a partition mapping, stating that partition x of the downstream asset depends on partitions x and x-1 of the upstream asset