https://dagster.io/ logo
#ask-ai
Title
# ask-ai
a

Axel Mendoza

02/23/2024, 3:21 PM
@Scout how to change this code to use in_process executor:
Copy code
yield RunRequest(
            run_key=run_record.dagster_run.run_id,  # avoid double firing for the same run
            run_config={},
        )
s

Scout

02/23/2024, 3:21 PM
To change the code to use the in_process executor, you would need to specify the executor configuration within the
run_config
dictionary. According to the documentation, to select the in_process executor, you should include the following top-level fragment in the config:
Copy code
yaml execution:   in_process: {}
However, since you are using the
RunRequest
object, you need to translate this YAML configuration into a Python dictionary. Here's how you can modify your code:
Copy code
python yield RunRequest(     run_key=run_record.dagster_run.run_id,  # avoid double firing for the same run     run_config={         "execution": {             "in_process": {}         }     }, )
This configuration tells Dagster to use the in_process executor when executing the run that is requested by this
RunRequest
.
a

Axel Mendoza

02/23/2024, 5:37 PM
@Scout with this:
Copy code
yield RunRequest(
            run_key=run_record.dagster_run.run_id,  # avoid double firing for the same run
            # TOEDIT: switch back to default after changing instance config.
            # run_config={},
            run_config={"execution": {"in_process": {}}},
        )
I get:
s

Scout

02/23/2024, 5:37 PM
It seems like your message was cut off and you didn't provide the outcome or error message you received after making the change to use the
in_process
executor. Could you please provide more details or the specific error message you encountered? This will help me assist you better.
a

Axel Mendoza

02/23/2024, 5:37 PM
@Scout with this:
Copy code
yield RunRequest(
            run_key=run_record.dagster_run.run_id,  # avoid double firing for the same run
            # TOEDIT: switch back to default after changing instance config.
            # run_config={},
            run_config={"execution": {"in_process": {}}},
        )
I get
Copy code
dagster._core.errors.DagsterInvalidConfigError: Error in config for job reconciliation_pipeline
    Error 1: Received unexpected config entry "in_process" at path root:execution. Expected: "{ config?: { in_process?: { marker_to_close?: String retries?: { disabled?: { } enabled?: { } } } multiprocess?: { max_concurrent?: Int? retries?: { disabled?: { } enabled?: { } } start_method?: { forkserver?: { preload_modules?: [String] } spawn?: { } } tag_concurrency_limits?: [{ key: String limit: Int value?: (String | { applyLimitPerUniqueValue: Bool }) }] } } }".
  File "/usr/local/lib/python3.10/site-packages/dagster/_grpc/impl.py", line 530, in get_external_execution_plan_snapshot
    create_execution_plan(
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/api.py", line 732, in create_execution_plan
    resolved_run_config = ResolvedRunConfig.build(job_def, run_config)
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/system_config/objects.py", line 169, in build
    raise DagsterInvalidConfigError(
s

Scout

02/23/2024, 5:37 PM
It appears that you've provided the same code snippet twice, and there is no error message or description of unexpected behavior included in your message. The code snippet you've posted is a Python generator yielding a
RunRequest
object with a specified
run_key
and
run_config
that uses the
in_process
executor. If you are encountering an issue with this code, please provide the error message or describe the unexpected behavior you are experiencing. Without additional context or error details, it's challenging to provide specific guidance on how to resolve the problem. If the code is not behaving as expected, here are some general debugging steps you can take: 1. Ensure that the
run_record
variable is defined and that
run_record.dagster_run.run_id
is the correct way to access the run ID. 2. Confirm that the
in_process
executor is correctly configured in your Dagster instance. The
in_process
executor should be available by default, but it's good to check if there are any instance-level configurations that might be affecting its behavior. 3. Check the logs for any error messages that might indicate what's going wrong. Dagster provides detailed logs that can help you understand the execution flow and pinpoint issues. 4. If you're running this within a larger Dagster pipeline, ensure that the rest of the pipeline is configured correctly and that there are no other issues that might be affecting the execution of this
RunRequest
. If you provide an error message or further details, I'll be better able to assist you.
The error message you're encountering indicates that there is an unexpected config entry "in_process" at the path
root:execution
in your
run_config
. The expected configuration structure seems to be slightly different than what you've provided. Based on the error message, it looks like the
in_process
configuration might need to be nested within a
config
key. However, the documentation does not provide a clear example of how to configure the
in_process
executor within a
RunRequest
. Given this, I would recommend checking the latest version of the Dagster documentation or any migration guides if there have been recent changes to the configuration schema. As a general debugging step, you can try adjusting the
run_config
to match the expected structure indicated by the error message. Here's an example of what this might look like, but please note that without explicit guidance from the documentation, this is speculative:
Copy code
python yield RunRequest(     run_key=run_record.dagster_run.run_id,  # avoid double firing for the same run     run_config={         "execution": {             "config": {                 "in_process": {}             }         }     }, )
If this does not resolve the issue, I would suggest looking for examples in the documentation that match the version of Dagster you are using, as configuration schemas can change between versions. If you are unable to find the necessary information in the documentation, consider reaching out to the Dagster community or support channels for further assistance.