@Scout how to change this code to use in_process executor:

yield RunRequest(
            run_key=run_record.dagster_run.run_id,  # avoid double firing for the same run
            run_config={},
        )

To change the code to use the in_process executor, you would need to specify the executor configuration within the

run_config

dictionary. According to the documentation, to select the in_process executor, you should include the following top-level fragment in the config:

yaml execution:   in_process: {}

However, since you are using the

RunRequest

object, you need to translate this YAML configuration into a Python dictionary. Here's how you can modify your code:

python yield RunRequest(     run_key=run_record.dagster_run.run_id,  # avoid double firing for the same run     run_config={         "execution": {             "in_process": {}         }     }, )

This configuration tells Dagster to use the in_process executor when executing the run that is requested by this

RunRequest

.

@Scout with this:

yield RunRequest(
            run_key=run_record.dagster_run.run_id,  # avoid double firing for the same run
            # TOEDIT: switch back to default after changing instance config.
            # run_config={},
            run_config={"execution": {"in_process": {}}},
        )

I get:

It seems like your message was cut off and you didn't provide the outcome or error message you received after making the change to use the

in_process

executor. Could you please provide more details or the specific error message you encountered? This will help me assist you better.

@Scout with this:

yield RunRequest(
            run_key=run_record.dagster_run.run_id,  # avoid double firing for the same run
            # TOEDIT: switch back to default after changing instance config.
            # run_config={},
            run_config={"execution": {"in_process": {}}},
        )

I get

dagster._core.errors.DagsterInvalidConfigError: Error in config for job reconciliation_pipeline
    Error 1: Received unexpected config entry "in_process" at path root:execution. Expected: "{ config?: { in_process?: { marker_to_close?: String retries?: { disabled?: { } enabled?: { } } } multiprocess?: { max_concurrent?: Int? retries?: { disabled?: { } enabled?: { } } start_method?: { forkserver?: { preload_modules?: [String] } spawn?: { } } tag_concurrency_limits?: [{ key: String limit: Int value?: (String | { applyLimitPerUniqueValue: Bool }) }] } } }".
  File "/usr/local/lib/python3.10/site-packages/dagster/_grpc/impl.py", line 530, in get_external_execution_plan_snapshot
    create_execution_plan(
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/api.py", line 732, in create_execution_plan
    resolved_run_config = ResolvedRunConfig.build(job_def, run_config)
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/system_config/objects.py", line 169, in build
    raise DagsterInvalidConfigError(

It appears that you've provided the same code snippet twice, and there is no error message or description of unexpected behavior included in your message. The code snippet you've posted is a Python generator yielding a

RunRequest

object with a specified

run_key

and

run_config

that uses the

in_process

executor. If you are encountering an issue with this code, please provide the error message or describe the unexpected behavior you are experiencing. Without additional context or error details, it's challenging to provide specific guidance on how to resolve the problem. If the code is not behaving as expected, here are some general debugging steps you can take: 1. Ensure that the

run_record

variable is defined and that

run_record.dagster_run.run_id

is the correct way to access the run ID. 2. Confirm that the

in_process

executor is correctly configured in your Dagster instance. The

in_process

executor should be available by default, but it's good to check if there are any instance-level configurations that might be affecting its behavior. 3. Check the logs for any error messages that might indicate what's going wrong. Dagster provides detailed logs that can help you understand the execution flow and pinpoint issues. 4. If you're running this within a larger Dagster pipeline, ensure that the rest of the pipeline is configured correctly and that there are no other issues that might be affecting the execution of this

RunRequest

. If you provide an error message or further details, I'll be better able to assist you.