Hi Dagster team. I am a bit stuck with an issue on...
# ask-community
Hi Dagster team. I am a bit stuck with an issue on partition assets and reconciliation sensor at the moment. I have an upstream job defined by one asset which is partionned using
. I have then a downstream asset linked to the upstream asset using
since I do not need to use IOManager in that case. All assets are selected under the
. However when I run one partition on the upstream asset I do not get any stale status on the downstream status and (of course) the sensor is not triggering. I am on dagster 1.1.14. I am missing something in my pipeline? Or is this feature not yet working?
cc @claire maybe
isn’t considered in the reconciliation sensor?
I should add that the pipeline works as expected if the upstream asset is not partionned.
I am tagging in case it gets forgotten. 🙂 Thanks for the support. @claire @jamie
Hi Greg.
are considered in the reconciliation sensor, so this likely isn't the core issue. Are all partitions of the upstream asset materialized?
My guess is that the downstream asset requires all upstream partitions to be materialized (as all upstream partitions are dependencies) so the reconciliation sensor won't kick off the downstream partitions until then.
Are all partitions of the upstream asset materialized?
No they are not. Your guess sounds very plausible. Any idea on how I could make this work such as the downstream assets would be materialized if any partition of the upstream asset is? @claire
What is the dependency relationship between the downstream asset and the upstream partitions? Is it the case that the downstream asset depends on all partitions of the upstream asset?
You could materialize all partitions of the upstream asset, and dagster will infer all dependencies of the downstream asset to be fulfilled. Thereafter, rematerializing any partition in the upstream asset will kick off a materialization of the downstream asset.
If you want to customize this behavior to materialize the downstream asset upon each upstream partition materialization regardless of if all partitions are materialized, you'll have to define a custom
that does this behavior for you: https://docs.dagster.io/concepts/partitions-schedules-sensors/asset-sensors#when-any-partitions-have-new-materializations
I ll look it up. Thanks @claire