https://dagster.io/ logo
#ask-community
Title
# ask-community
g

Greg

02/02/2023, 1:06 PM
Hi Dagster team. I am a bit stuck with an issue on partition assets and reconciliation sensor at the moment. I have an upstream job defined by one asset which is partionned using
DailyPartitionsDefinition
. I have then a downstream asset linked to the upstream asset using
non_argument_deps
since I do not need to use IOManager in that case. All assets are selected under the
asset_reconciliation_sensor
. However when I run one partition on the upstream asset I do not get any stale status on the downstream status and (of course) the sensor is not triggering. I am on dagster 1.1.14. I am missing something in my pipeline? Or is this feature not yet working?
j

jamie

02/02/2023, 3:57 PM
cc @claire maybe
non_argument_deps
isn’t considered in the reconciliation sensor?
g

Greg

02/02/2023, 4:07 PM
I should add that the pipeline works as expected if the upstream asset is not partionned.
I am tagging in case it gets forgotten. 🙂 Thanks for the support. @claire @jamie
c

claire

02/06/2023, 7:43 PM
Hi Greg.
non_argument_deps
are considered in the reconciliation sensor, so this likely isn't the core issue. Are all partitions of the upstream asset materialized?
My guess is that the downstream asset requires all upstream partitions to be materialized (as all upstream partitions are dependencies) so the reconciliation sensor won't kick off the downstream partitions until then.
g

Greg

02/06/2023, 7:46 PM
Are all partitions of the upstream asset materialized?
No they are not. Your guess sounds very plausible. Any idea on how I could make this work such as the downstream assets would be materialized if any partition of the upstream asset is? @claire
c

claire

02/06/2023, 7:52 PM
What is the dependency relationship between the downstream asset and the upstream partitions? Is it the case that the downstream asset depends on all partitions of the upstream asset?
You could materialize all partitions of the upstream asset, and dagster will infer all dependencies of the downstream asset to be fulfilled. Thereafter, rematerializing any partition in the upstream asset will kick off a materialization of the downstream asset.
If you want to customize this behavior to materialize the downstream asset upon each upstream partition materialization regardless of if all partitions are materialized, you'll have to define a custom
multi_asset_sensor
that does this behavior for you: https://docs.dagster.io/concepts/partitions-schedules-sensors/asset-sensors#when-any-partitions-have-new-materializations
g

Greg

02/06/2023, 7:58 PM
I ll look it up. Thanks @claire