Experimenting with Dagster, so another question 🙂 I tried auto-materialization with partitioned assets. When the latest partition runs, downstream tasks are also initiated. However, any of the previous partitions are run, nothing happens in the downstream jobs. Is it a bug or the expected behaviour?

Hi @Sabri Karagönen! what version of dagster are you on? in the most recent version of dagster we updated the behavior in a way that should match what you're expecting (materializations of older partitions should result in materializations of downstream assets)

Hi Owen, It’s 1.3.13

we just tried this with 1.3.14 and it has the same behavior, should we create an issue for this with reproduction?

ah and one more thing to add, there's a default "max_materializations_per_minute" of 1, so if all of those asset1_auto materializations came within a short time period, then I'd expect only one partition of the downstream to be materialized -- is it possible that's what's happening? If you go to the "asset details" page for the downstream assets, you should be able to see a history of reasons that partitions were/were not kicked off

hmm, this could be the reason indeed, but isn’t that a bug then? because what I’d expect is that the upstream is materialized, I have data there, my downstream depends on that data, so my downstream should be materialized as well, whether or not it happens in 1min or it is spread into 7mins (because 7 partitions) is not a concern for me, but I expect the system to eventually converge

that's a reasonable interpretation -- the reason it's set up this way is to prevent small actions (for example, adding a new hourly-partitioned asset with thousands of missing partitions) from automatically kicking off a bunch of work. It's possible to increase this limit (or remove it all together, by setting

AutoMaterializePolicy.eager(max_materializations_per_minute=None)

) if that works better for your usecase

in that case would kicking these off only after backfill runs be a reasonable way to handle this? e.g. if I am running a backfill for a period where there are downstream dependencies, I probably want them to be materialized as well. Alternatively, a UI option during a backfill could help me with this

would it be reasonable to include those downstream dependencies in the backfill selection as well? or is that what you're referencing with the "UI option during a backfill" bit -- something that says "include all downstreams as well" or something of the sort

apologies, missed this message. yeah. that’s what I was referring to. Ideally, I’d expect to have “all downstreams” selected by default, and I’d also like to be able to selectively update certain assets, sort of like a checklist that I can search and filter.