I wanted to understand if there’s an example where...
# ask-community
I wanted to understand if there’s an example where assets and ops interplay? For example, a collection of ops is what leads to the creation of the asset. Some pipeline I’m importing is using assets, and another is using ops. Whats the best way to combine them? Also, with AssetGroups, how do i get it to turn into a graph instead of a job? Thanks!
hi @Charles Leung! For the first part of the question, today's release will contain preliminary support for "graph backed assets", which allow you define a graph of ops which work to produce a single (or multiple) assets. This looks something like
Copy code
from dagster import graph, AssetsDefinition

def my_graph(foo, bar):
    ... # normal graph stuff
    return asset_producing_op(...)

my_graph_backed_asset = AssetsDefintion.from_graph(my_graph)
❤️ 3
for your next question, would the ideal end state be that both your pipelines get combined into a single job? or is it better for them to stay separate (and just have one run after the other)? If they're meant to stay separate, AssetSensors would probably be a reasonable pattern here (launching the op-based pipeline after it detects that a specific asset has been materialized)
For turning an AssetGroup into a graph instead of a job, what's the usecase you're intending for that? JobDefinitions do have a reference to their underlying GraphDefinition, so I suppose you could do something like
my_graph = AssetGroup.build_job(...).graph
, but I wouldn't necessarily recommend that depending on the scenario
At least for me, the main thing it gets me is the ability to reuse ops across assets
Hey owen! thanks for the information! Our current pattern is designing an end to end graph that composes of other graphs. With jobs, i’m not too sure how that would work but with a .graph i suppose i could then create a graph that generates assetgroups 🙂 . Your solution helps! Let me know if I’m doing some kind of anti-pattern however 😆
the main issue with that solution (and it's a pretty big one 😛) is that it wipes out the asset-information (as that's stored at the job level at the moment). This means that running it will not result in AssetMaterializations being generated, nor will those assets show up in dagit. I think the ideal would just be to combine everything into one big asset group if possible
Something to add here is that it's possible to add asset groups together using the
operator. In a sense, that's the equivalent of combining multiple graphs into a single mega-graph.