Are there plans to support dynamically partitioned Assets in the future or a way to do the following with current features? In essence, I need to process experiment results and would like to use partitioned assets for that. Every few days (inconsistent) an experiment result will be made available for Dagster to process, and I would like to handle each result as a separate partition. The list of partitions would have to grow dynamically and be re-fetched before kicking off runs for missing partitions. Apologies if the feature already exists, the docs lead me to believe it doesn’t, as the

@dynamic_partitioned_config

is under “Partitioned non-Asset Jobs”.

Hi Vinnie. Dynamically partitioned assets are definitely something we've considered and eventually would like to support it. Other users have spoken up about this and we're aware this is a fairly common use case for SDAs. I'm not sure when we'll get around to supporting this, but you can use this issue to track: https://github.com/dagster-io/dagster/issues/7943

Hi Claire, thanks for getting back. In this case I’ll have to stick to jobs for now.

I am facing a similar use case. We have external service that generate datasets, and the dagster pipeline is to pick up each dataset and run through a set of ops. With this feature, we could take advantage of the software-defined asset, but for now we are stuck with the op-oriented approach.

@Vinnie @Zhiyuan Ma - we're now prioritizing first-class support for use cases like the ones you brought up and gathering detailed requirements. Would either of you be open to a short call to talk about what you're trying to do?

Yes! Let me know when you want to talk. I've got my use case implemented with the op oriented approach, but certainly it would be good to leverage the software defined assets capabilities.

Awesome - I'll DM you

Always! Let’s find something next week :)