https://dagster.io/ logo
#random
Title
# random
a

Akshay Verma

02/28/2024, 8:25 AM
How can I export a PNG/SVG of my Dagster pipeline programmatically?
s

Selim Raboudi

02/28/2024, 11:16 AM
ask your question on the #ask-ai channel, you are more likely to get an answer there 🙂
👍 1
s

Stephen Snyder

03/05/2024, 3:03 PM
I'm looking into this as well; my case is graphs of SW-defined assets; looks like I can access a representation of the graph once it's been loaded into a Definitions() class via defn.get_asset_graph(), which returns a nested dict of upstream and downstream edges by asset key. So probably a quick few steps after that to load into something like networkx, label based on depth, and then plot as a multipartite layout. I'll try and circle back later this week or next if I have time.
a

Akshay Verma

03/05/2024, 3:23 PM
Great, @Stephen Snyder thanks for the message. Look forward to hearing from you.
s

Stephen Snyder

03/05/2024, 3:29 PM
something like this:
Copy code
from dagster import (Definitions, 
                     define_asset_job, 
                     load_assets_from_modules
                        )
import networkx as nx

'''
in my case I am loading SW-defined assets from modules
'''

defs = Definitions(
            #your SW-defined assets+job here
        )

graph = defs.get_asset_graph()

G = nx.DiGraph()

for node,links in graph.asset_dep_graph['downstream'].items():
    G.add_node(node)
    G.add_edges_from([(node,link) for link in links])

# find root nodes with no upstream parents
root_nodes = [node for node in G.nodes() if G.in_degree(node) == 0]

for rt in root_nodes:
    G.nodes[rt]['depth'] = 1
    descendants = nx.descendants(G,rt)
    for desc in descendants:
        G.nodes[desc]['depth'] = nx.shortest_path_length(G,rt,desc) + 1

pos = nx.multipartite_layout(G,subset_key='depth')
nx.draw(G,pos=pos, with_labels=True, font_weight='bold')
then you can decide if you want to use matplotlib and figure save options to push it out to PNG, or if you want to make your svg directly from the networkx graph + position layout using something like pyCairo
a

Akshay Verma

03/05/2024, 3:32 PM
Thanks a lot. Is very helpful
🎉 1