@Scout What is the use case for a

in_process_executor

?

The

in_process_executor

is an executor in Dagster that executes all steps of a job within a single process. The use case for the

in_process_executor

includes scenarios where you want to: 1. Simplify debugging by running the entire job within a single process, which can make it easier to trace and understand the flow of execution and troubleshoot issues. 2. Avoid the overhead associated with starting multiple processes, which can be beneficial for jobs with short-running steps where the overhead of process creation might outweigh the benefits of parallel execution. 3. Work with data or compute steps that are not easily serializable across process boundaries, as the

in_process_executor

does not require serialization of data between steps. 4. Run jobs in environments where multiprocessing is not desired or supported, such as certain restricted or constrained execution environments. The documentation also notes that when using the

execute_in_process

method to execute a job, the

executor_def

on the job will be ignored and replaced with the in-process executor. This means that even if the job is defined with a different executor, using

execute_in_process

will ensure that the job runs within a single process. Additionally, when specifying the

in_process

configuration for the default

multi_or_in_process_executor

, you can switch the execution mode of the job to run in-process rather than in multiprocess mode. This can be done via the job's run configuration. In summary, the

in_process_executor

is useful for scenarios where single-process execution is preferred for simplicity, debugging, performance considerations, or compatibility with the execution environment.

@Scout If I need the ops defined in the job below to operate in sequence, do I need to use an in_process_executor?

@job(
    name='deploy_job',
    resource_defs=job_resources(),
    executor_def=in_process_executor,
    config=RunConfig(
        ops={'load_op': app_db_load_config,
             'assemble_op': app_db_load_config,
             'promote_op': app_db_load_config, }
    ),
)
def utilizations_app_deploy_job():
    load_data = load_op()
    assemble_data = assemble_op(upstream=load_data)
    promote_data = promote_op(upstream=assemble_data)
    refresh_partitions = refresh_partitions_op(upstream=promote_data)
    refresh_partitions = refresh_partitions_op(upstream=refresh_partitions)
    post_deploy_tasks_op(upstream=refresh_partitions)

In Dagster, the execution order of ops within a job is determined by their dependencies, not by the executor used. The

in_process_executor

is not required to run ops in sequence; it is the dependencies between ops that enforce the execution order. In the job definition you provided, the ops are already set up to run in sequence due to the

upstream

arguments, which create dependencies between the ops: 1.

load_data

will run first because it has no dependencies. 2.

assemble_data

will run after

load_data

because it depends on

load_data

. 3.

promote_data

will run after

assemble_data

because it depends on

assemble_data

. 4.

refresh_partitions

will run after

promote_data

because it depends on

promote_data

. 5.

post_deploy_tasks_op

will run last because it depends on

refresh_partitions

. The

executor_def=in_process_executor

in the job definition is specifying that the job should use the in-process executor, but this is not what enforces the sequential execution. It is the dependencies that you have defined between the ops that ensure they will run in the specified order. If you want to ensure that the ops run in the same process for reasons such as shared memory or debugging, then specifying

in_process_executor

is appropriate. However, if your only goal is to have the ops run in sequence, then you simply need to define the dependencies correctly, as you have done, and you can use the default executor or any other executor that respects the dependencies.