Is there a way to kick off a local Dagster job within a Python program and still use Python multiprocessing? We have some integration tests that do one or more pipeline runs, as well as some CLI entry points, all of which I'd like to preserve while migrating to Dagster. I found

DagsterInstance

but it looks like it's for internal use. What I'm doing now is providing a wrapper like this:

def run_pipeline(...):
    with tempfile.NamedTemporaryFile(mode="w", suffix=".yaml") as temp_config_file:
        shared_config = {
            "a_config_option": x,
            "another_config_option": y,
        }
        yaml.dump(
            {"resources": {"shared_config": {"config": shared_config}}},
            temp_config_file,
        )
        temp_config_file.flush()

        subprocess.check_call([
            "dagster",
            "job",
            "execute",
            "-m", "my.dagster_repo.module",
            "-j", "my_job_name",
            "-c", temp_config_file.name,
        ])

It would be more convenient to just call this directly in Python. Ideally including

dagster.yaml

settings too would be great, so the wrapper script can set some of those options.

its a bit cumbersome at the moment, but looks like this:

result = execute_pipeline(
  reconstructable(my_job_name),
  instance=DagsterInstance.get(),
)

Thanks! (Found execute_pipeline docs here .) Searching for "run job" got me to https://docs.dagster.io/_apidocs/execution#executing-jobs , and the intro docs got me to https://docs.dagster.io/getting-started#running-the-job-programmatically, maybe a note in the latter pointing to

execute_pipeline

would be helpful?

@Dagster Bot discussion Execute job with multiprocessing in Python

Question in the thread has been surfaced to GitHub Discussions for future discoverability: https://github.com/dagster-io/dagster/discussions/8467