I'm calling the graphql endpoint over HTTP with a

launchPipelineExecution

mutation to start a pipeline. Looking to have it start in process on the dagit instance. I can't figure out the new 0.8 selector I need to specify. Here's what I got so far:

launchPipelineExecution(
	executionParams: {
		selector: {
			repositoryLocationName: "<<in_process>>",
			repositoryName: "my_repository",
			pipelineName: "my_pipeline",
		},
		mode: "default",
	}
)

Getting this error:

RESPONSE >>> {'data': {'launchPipelineExecution': {'__typename': 'PipelineNotFoundError', 'message': 'Could not find Pipeline <<in_process>>.my_repository.my_pipeline', 'pipelineName': 'my_pipeline'}}}

I'm trying to copy how the dagster_graphql test suite does it but not succeeding. On a side note, might be useful to incorporate "execute pipelines over graphql" machinery into the main dagster_graphql API, because it's very useful for use cases like mine where I have pipelines launching other pipelines, and it's a shame all that useful client code is buried in tests.

ya this stuff is still very rough The best thing to do is in your

workspace.yaml

you can specify

location_name

on your

load_from

targets and then you use that name for

repositoryLocationName

@alex worked, thank you! 🙏

If you want some context on the architectural changes that these rough edges fell out from they are covered best in this video

Watched it, very informative

oh you can have multiple - just no way to bound how many

Actually I hacked something up for this in 0.7:

def wait_for_run_slot(pipeline_name, max_runs=1):

    instance = DagsterInstance.get()

    def get_active_runs(pipeline_name):
        started_runs = len(instance.get_runs(PipelineRunsFilter(pipeline_name=pipeline_name, status=PipelineRunStatus.STARTED)))
        not_started_runs = len(instance.get_runs(PipelineRunsFilter(pipeline_name=pipeline_name, status=PipelineRunStatus.NOT_STARTED)))
        return started_runs + not_started_runs

    while get_active_runs(pipeline_name) >= max_runs:
        time.sleep(2)

This worked nicely when called in the parent pipeline:

wait_for_run_slot("child_pipeline", 10)

oh great ya i was just thinking through how to suggest something like this

Thx

hmmm - you are not calling

executeRunInProcess

at all right? and you are just using the default run launcher?

I don't even have a way to verify whether my parent and children pipelines are in fact being run in parallel

just to verify you see

CliApiRunLauncher

if you look at the “Instance Details” in

dagit

Yup

Run Launcher:
     module: dagster.core.launcher.cli_api_run_launcher
     class: CliApiRunLauncher
     config:
       {}

and dagit locks up - very peculiar

Yes, until the pipelines complete. Then I can review everything and see no errors in the pipeline results.

Can you describe in more detail what you are seeing? Does the web page not load at all ? Just certain views?

Let me gather some more clues and get back

appreciated

Looks like it was an issue on my end, dagit is fine. My laptop kept freezing up while my parent pipeline spawned a whole bunch of children, and with each child being a noop that returned immediately (hacking on a 0.8 prototype currently) it was constantly spawning new ones. Looked like dagit was dead, but putting a sleep into the children revealed it was fine. My bad!