Hello, I have a question about using Dynamic Outpu...
# ask-community
y
Hello, I have a question about using Dynamic Outputs. Can the downstream operations be assets or do they need to be ops? I have a set of downstream assets that need to be generated per
metric
, and I'd like to input the metric names in a config so that
metric_group
will fan out per
metric
. Or can outputs of
multi_assets
be dynamic? Thanks!
j
Hi @Yang i'm not sure if the discussion in this thread matches your use case entirely, but it may be helpful https://dagster.slack.com/archives/C01U954MEER/p1661402901092569 one thing you might consider is manually yielding
AssetMaterialization
s from the ops at the end of your graph https://docs.dagster.io/concepts/assets/asset-materializations we don't have full support for tracking lineage of
AssetMaterializations
yielded in this way, but we have an issue we're tracking for it here
y
Hmm! I'll check these out. I haven't written the ETL yet. Maybe it's easier just to write ops?
Oh wait, for using Dynamic Outputs, what if all the dynamic outputs also depend on another output of the same upstream op or asset? It looks like downstream ops can only take context, and one thing passed by
yield DynamicOutput
Maybe I do want to use Partitions
I think I'd like to be able to define the partition keys for a
@static_partitioned_config
in a config.
Is the setup for a static partition the same for assets as ops? Thanks!
j
if you need the downstream op to take an additional arg you can follow this example with a lambda function https://docs.dagster.io/concepts/ops-jobs-graphs/dynamic-graphs#additional-arguments . setting up a partition for assets vs ops has slightly different syntax. for an asset you would define a
StaticPartitionDefinition
and pass it to the asset decorator https://docs.dagster.io/concepts/partitions-schedules-sensors/partitions#partitioned-assets
y
Oh! Ok great, thanks. Just to make sure, if I use ops, all the steps in the job needs to be an op? I can't have a static asset as the additional argument? If I ops, then there is one that I'd like to reuse for another job that uses all assets currently. Can ops be part of an asset graph or is there an easy way to convert an op to an asset? Thanks!
j
yeah if you use ops, all the steps in the job need to be ops. If you want to use an op in an asset graph, you could use the graph backed assets concept https://docs.dagster.io/concepts/assets/software-defined-assets#graph-backed-assets
y
Oh I see...Ok, I think I can do that thanks!
There's no way to make a dynamic asset right now, is there? In other words, I would want a multi-asset, but dynamically determine the outputs (I wouldn't define them up front bc it depends on the run)
j
yeah that's not something we currently support
y
gotcha