https://dagster.io/ logo
Join Slack
Powered by
```Exception: Error in config for job Error 1:...
# ask-community
t
Copy code
Exception: Error in config for job
    Error 1: Received unexpected config entry "in_process" at path root:execution. Expected: "{ config: { cluster: { existing: { address: (String | { env: String }) } kube?: { } local?: { } lsf?: { } moab?: { } oar?: { } pbs?: { } sge?: { } slurm?: { } ssh?: { } yarn?: { } } } }".
    Error 2: Missing required config entry "config" at path root:execution. Sample config for missing entry: {'config': {'cluster': '<selector>'}}
k
Hi. Be it legacy or latest version, You need to provide an executor in the config. Please refer to this documentation. 🙂
t
I'm currently providing dask_executor to my job via the executor_def parameter
Then i'm filling the config but i can't put my dask executor in the yaml with somethin like that
Copy code
execution:
  dask:
    config:
      cluster:
        pbs:
          queue: mpi_4
          project: dagster
          walltime: '00:05:00'
          memory: '120gb'
          cores: 28
          resource_spec: 'select=4:ncpus=28:mem=120gb'
          n_workers: 1
I can only do this
Copy code
execution: 
  config:
    cluster:
      pbs:
        cores: 28
        memory: 120gb
        n_workers: 1
        project: dagster
        queue: mpi_4
        resource_spec: "select=4:ncpus=28:mem=120gb"
        walltime: "00:05:00"
c
Just to make sure I understand, you're getting this error:
Copy code
Exception: Error in config for job
    Error 1: Received unexpected config entry "in_process" at path root:execution. Expected: "{ config: { cluster: { existing: { address: (String | { env: String }) } kube?: { } local?: { } lsf?: { } moab?: { } oar?: { } pbs?: { } sge?: { } slurm?: { } ssh?: { } yarn?: { } } } }".
    Error 2: Missing required config entry "config" at path root:execution. Sample config for missing entry: {'config': {'cluster': '<selector>'}}
When this config is used:
Copy code
execution: 
  config:
    cluster:
      pbs:
        cores: 28
        memory: 120gb
        n_workers: 1
        project: dagster
        queue: mpi_4
        resource_spec: "select=4:ncpus=28:mem=120gb"
        walltime: "00:05:00"
Is that correct?
t
Yes it is correct
And i'm using dask executor, in legacy version i was able to specify it in the config
like that
Copy code
execution: dask:
  config:
    cluster:
      pbs:
        cores: 28
        memory: 120gb
        n_workers: 1
        project: dagster
        queue: mpi_4
        resource_spec: "select=4:ncpus=28:mem=120gb"
        walltime: "00:05:00"
c
Okay I think I found the source of the error. Could've sworn this was fixed, but apparently not.
You can track progress on a fix here
t
thank you, i will try this 🙂
but i have to use the config with 
dask :
in it between 
execution :
and `config :`or the config without ?
c
hey: I don't think there's a solution for you right now. This config should be working:
Copy code
execution: 
  config:
    cluster:
      pbs:
        cores: 28
        memory: 120gb
        n_workers: 1
        project: dagster
        queue: mpi_4
        resource_spec: "select=4:ncpus=28:mem=120gb"
        walltime: "00:05:00"
Since there's only one executor on a job, there's no need for the 
dask
key: https://docs.dagster.io/guides/dagster/graph_job_op#executors
k
Copy code
execution: 
  config:
    cluster:
      pbs:
        cores: 28
        memory: 120gb
        n_workers: 1
        project: dagster
        queue: mpi_4
        resource_spec: "select=4:ncpus=28:mem=120gb"
        walltime: "00:05:00"
Hi. The above config will work if you do not pass dask_executor in the 
@job
decorator parameters. Since you are saying that you have mentioned it there, for now you will have to use 
dask:
between 
execution:
and 
config:
as it will be looking for the config. :-)
t
@Keshav ( I need the decorator 
executor_def=dask_executor
on my job ) I can't put the config with 
dask :
in the interface then run it ( the button is disabled ), maybe you have a javascript code that check config :
But i will try directly with dagster ( i made the change code made by Chris in my environnement )
Same error with dagster
Copy code
(/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11) semexp@datarmor3:/home1/datawork/semexp/workspace/dagit/semaphore-dagster-config/pipelines> dagster job execute -f template_runner.py -c template_runner.yaml
Traceback (most recent call last):
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/bin/dagster", line 8, in <module>
    sys.exit(main())
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/dagster/cli/__init__.py", line 50, in main
    cli(auto_envvar_prefix=ENV_PREFIX)  # pylint:disable=E1123
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/dagster/cli/job.py", line 128, in job_execute_command
    execute_execute_command(instance, kwargs, True)
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/dagster/core/telemetry.py", line 84, in wrap
    result = f(*args, **kwargs)
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/dagster/cli/pipeline.py", line 374, in execute_execute_command
    result = do_execute_command(pipeline, instance, config, mode, tags, solid_selection, preset)
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/dagster/cli/pipeline.py", line 565, in do_execute_command
    return execute_pipeline(
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/dagster/core/execution/api.py", line 373, in execute_pipeline
    return _logged_execute_pipeline(
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/dagster/core/telemetry.py", line 84, in wrap
    result = f(*args, **kwargs)
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/dagster/core/execution/api.py", line 415, in _logged_execute_pipeline
    pipeline_run = instance.create_run_for_pipeline(
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/dagster/core/instance/__init__.py", line 750, in create_run_for_pipeline
    execution_plan = create_execution_plan(
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/dagster/core/execution/api.py", line 743, in create_execution_plan
    resolved_run_config = ResolvedRunConfig.build(pipeline_def, run_config, mode=mode)
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/dagster/core/system_config/objects.py", line 155, in build
    raise DagsterInvalidConfigError(
dagster.core.errors.DagsterInvalidConfigError: Error in config for job
    Error 1: Received unexpected config entry "dask" at path root:execution. Expected: "{ config: { cluster: { existing: { address: (String | { env: String }) } kube(/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11) semexp@datarmor3:/home1/datawork/semexp/workspace/dagit/semaphore-dagster-config/pipelines>     Error 2: Missing required config entry "config" at path root:execution. Sample config for missing entry: {'config': {'cluster': '<selector>'}}
without the 
dask
executor and the 
dask :
in the config, it's better but i need dask :(
3 Views
Open in Slack
PreviousNext
Powered by