Hello, I have a question about using Dynamic Outpu...
# ask-community
Hello, I have a question about using Dynamic Outputs. Can the downstream operations be assets or do they need to be ops? I have a set of downstream assets that need to be generated per
, and I'd like to input the metric names in a config so that
will fan out per
. Or can outputs of
be dynamic? Thanks!
Hi @Yang i'm not sure if the discussion in this thread matches your use case entirely, but it may be helpful https://dagster.slack.com/archives/C01U954MEER/p1661402901092569 one thing you might consider is manually yielding
s from the ops at the end of your graph https://docs.dagster.io/concepts/assets/asset-materializations we don't have full support for tracking lineage of
yielded in this way, but we have an issue we're tracking for it here
Hmm! I'll check these out. I haven't written the ETL yet. Maybe it's easier just to write ops?
Oh wait, for using Dynamic Outputs, what if all the dynamic outputs also depend on another output of the same upstream op or asset? It looks like downstream ops can only take context, and one thing passed by
yield DynamicOutput
Maybe I do want to use Partitions
I think I'd like to be able to define the partition keys for a
in a config.
Is the setup for a static partition the same for assets as ops? Thanks!
if you need the downstream op to take an additional arg you can follow this example with a lambda function https://docs.dagster.io/concepts/ops-jobs-graphs/dynamic-graphs#additional-arguments . setting up a partition for assets vs ops has slightly different syntax. for an asset you would define a
and pass it to the asset decorator https://docs.dagster.io/concepts/partitions-schedules-sensors/partitions#partitioned-assets
Oh! Ok great, thanks. Just to make sure, if I use ops, all the steps in the job needs to be an op? I can't have a static asset as the additional argument? If I ops, then there is one that I'd like to reuse for another job that uses all assets currently. Can ops be part of an asset graph or is there an easy way to convert an op to an asset? Thanks!
yeah if you use ops, all the steps in the job need to be ops. If you want to use an op in an asset graph, you could use the graph backed assets concept https://docs.dagster.io/concepts/assets/software-defined-assets#graph-backed-assets
Oh I see...Ok, I think I can do that thanks!
There's no way to make a dynamic asset right now, is there? In other words, I would want a multi-asset, but dynamically determine the outputs (I wouldn't define them up front bc it depends on the run)
yeah that's not something we currently support