Hey Team, is it possible to create the graph in re...
# ask-community
s
Hey Team, is it possible to create the graph in repo.py( dagster-user-code) dynamically based on the request?. Currently i am able to do it for static jsons. Note: Dagster is deployed using helm on k8
Copy code
def example_graph(request_json):
    graph_def = GraphDefinition(
        name="step_isolated_job",
        node_defs=get_op_list(proxy_json),
        dependencies=get_dg(proxy_json)
    )
    return graph_def
request_json={"ops":{"node1":d}}
step_isolated_job = example_graph(request_json).to_job(
        name="step_isolated_job",
        resource_defs={
            **RESOURCES_LOCAL
        },
        executor_def=k8s_job_executor,
      
    )
d
Hi Sundara - there's no need to double-post or delete and re-post, if you ask a question on the weekend it sometimes will take until Monday for it get answered. We don't currently support varying the structure of a Dagster job on each run or on each request. The way to tell Dagster to reload your code from your pipeline yaml is to press the Reload button in Dagit on the Workspace tab, or make a reloadRepositoryLocation call using our graphql API: https://docs.dagster.io/_apidocs/libraries/dagster-graphql#dagster_graphql.DagsterGraphQLClient.reload_repository_location
s
is there any way to achieve this using the current dagster versions??
Like on each request, using different pipeline.ymls or any way to send a json to the repository (so that i can create graphs based on that)
d
The only way im aware of doing this in dagster currently is by manually reloading the location before launching each run when you want the structure of the job to change
Or via API using the call I mentioned above
s
I hope the above is for updating the repo. But what we need is just the graph updation inside that repo.
d
That's right - I think what you're asking for isn't currently possible in Dagster, other than more constrained map/collect features: https://docs.dagster.io/concepts/ops-jobs-graphs/dynamic-graphs#dynamic-graphs
s
Copy code
# create the op which submit the job
default_config = {"ops": {"do_something": {"config": {"config_param": "stuff"}}}}
def example_graph(request_json):
    graph_def = GraphDefinition(
        name="step_isolated_job",
        node_defs=get_op_list(request_json),
        dependencies=get_dg(request_json)
    )
    return graph_def

#config_schema=dict
@op(config_schema={"do_something": {} })
def graph_job(context):
    # create the graph
    <http://context.log.info|context.log.info>("starting the graph creation")
    options=context.solid_config.get("data")
    step_isolated_job = example_graph(options)
    <http://context.log.info|context.log.info>(step_isolated_job)
    <http://context.log.info|context.log.info>("stopped the graph creation")
    return step_isolated_job

@job(config=default_config)
def simple_job():
    graph_job().to_job(
        name="step_isolated_job",
        resource_defs={
            **RESOURCES_LOCAL
        },
        executor_def=k8s_job_executor,
       # config=pipeline_config
    )
  
@repository
def example_repo():
    return [simple_job()]
Will this not work?
d
I see an dagster op that returns a graph there - that's not something that's supported in dagster, no
graphs are defined at definition time when the code is loaded in dagit, and ops run at the time that a run is launched
s
Okay, which means currently there is no way to change the graph structure in dagster-user-code repository.
d
The only to do it currently would be with the workaround i mentioned - making a reloadRepositoryLocation call when you know the config has changed and want to update the graph structure. There isn't a way to vary it using request configuration on each run in the way that I think you were hoping to do.
This is our example for how to create a graph from a YAML file or some other external data source: https://docs.dagster.io/concepts/ops-jobs-graphs/jobs-graphs#graph-dsl That is possible, but there isn't a way to pass in config that is computed on each run