https://dagster.io/ logo
Powered by
Title
t

Thomas Mignon

12/01/2021, 9:12 AM
Exception: Error in config for job
    Error 1: Received unexpected config entry "in_process" at path root:execution. Expected: "{ config: { cluster: { existing: { address: (String | { env: String }) } kube?: { } local?: { } lsf?: { } moab?: { } oar?: { } pbs?: { } sge?: { } slurm?: { } ssh?: { } yarn?: { } } } }".
    Error 2: Missing required config entry "config" at path root:execution. Sample config for missing entry: {'config': {'cluster': '<selector>'}}
k

Keshav

12/01/2021, 1:00 PM
Hi. Be it legacy or latest version, You need to provide an executor in the config. Please refer to this documentation. 🙂
t

Thomas Mignon

12/01/2021, 3:22 PM
I'm currently providing dask_executor to my job via the executor_def parameter
Then i'm filling the config but i can't put my dask executor in the yaml with somethin like that 
execution:
  dask:
    config:
      cluster:
        pbs:
          queue: mpi_4
          project: dagster
          walltime: '00:05:00'
          memory: '120gb'
          cores: 28
          resource_spec: 'select=4:ncpus=28:mem=120gb'
          n_workers: 1
I can only do this
execution: 
  config:
    cluster:
      pbs:
        cores: 28
        memory: 120gb
        n_workers: 1
        project: dagster
        queue: mpi_4
        resource_spec: "select=4:ncpus=28:mem=120gb"
        walltime: "00:05:00"
c

chris

12/01/2021, 4:16 PM
Just to make sure I understand, you're getting this error: 
Exception: Error in config for job
    Error 1: Received unexpected config entry "in_process" at path root:execution. Expected: "{ config: { cluster: { existing: { address: (String | { env: String }) } kube?: { } local?: { } lsf?: { } moab?: { } oar?: { } pbs?: { } sge?: { } slurm?: { } ssh?: { } yarn?: { } } } }".
    Error 2: Missing required config entry "config" at path root:execution. Sample config for missing entry: {'config': {'cluster': '<selector>'}}
When this config is used: 
execution: 
  config:
    cluster:
      pbs:
        cores: 28
        memory: 120gb
        n_workers: 1
        project: dagster
        queue: mpi_4
        resource_spec: "select=4:ncpus=28:mem=120gb"
        walltime: "00:05:00"
Is that correct?
t

Thomas Mignon

12/01/2021, 4:49 PM
Yes it is correct
And i'm using dask executor, in legacy version i was able to specify it in the config
like that
execution: dask:
  config:
    cluster:
      pbs:
        cores: 28
        memory: 120gb
        n_workers: 1
        project: dagster
        queue: mpi_4
        resource_spec: "select=4:ncpus=28:mem=120gb"
        walltime: "00:05:00"
c

chris

12/01/2021, 8:07 PM
Okay I think I found the source of the error. Could've sworn this was fixed, but apparently not.
You can track progress on a fix here
t

Thomas Mignon

12/01/2021, 9:50 PM
thank you, i will try this 🙂
but i have to use the config with 
dask :
in it between 
execution :
and `config :`or the config without ?
c

chris

12/01/2021, 10:41 PM
hey: I don't think there's a solution for you right now. This config should be working: 
execution: 
  config:
    cluster:
      pbs:
        cores: 28
        memory: 120gb
        n_workers: 1
        project: dagster
        queue: mpi_4
        resource_spec: "select=4:ncpus=28:mem=120gb"
        walltime: "00:05:00"
Since there's only one executor on a job, there's no need for the 
dask
key: https://docs.dagster.io/guides/dagster/graph_job_op#executors
k

Keshav

12/02/2021, 3:12 PM
execution: 
  config:
    cluster:
      pbs:
        cores: 28
        memory: 120gb
        n_workers: 1
        project: dagster
        queue: mpi_4
        resource_spec: "select=4:ncpus=28:mem=120gb"
        walltime: "00:05:00"
Hi. The above config will work if you do not pass dask_executor in the 
@job
decorator parameters. Since you are saying that you have mentioned it there, for now you will have to use 
dask:
between 
execution:
and 
config:
as it will be looking for the config. :-)
t

Thomas Mignon

12/02/2021, 6:31 PM
@Keshav ( I need the decorator 
executor_def=dask_executor
on my job ) I can't put the config with 
dask :
in the interface then run it ( the button is disabled ), maybe you have a javascript code that check config :
But i will try directly with dagster ( i made the change code made by Chris in my environnement )
Same error with dagster
(/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11) semexp@datarmor3:/home1/datawork/semexp/workspace/dagit/semaphore-dagster-config/pipelines> dagster job execute -f template_runner.py -c template_runner.yaml
Traceback (most recent call last):
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/bin/dagster", line 8, in <module>
    sys.exit(main())
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/dagster/cli/__init__.py", line 50, in main
    cli(auto_envvar_prefix=ENV_PREFIX)  # pylint:disable=E1123
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/dagster/cli/job.py", line 128, in job_execute_command
    execute_execute_command(instance, kwargs, True)
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/dagster/core/telemetry.py", line 84, in wrap
    result = f(*args, **kwargs)
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/dagster/cli/pipeline.py", line 374, in execute_execute_command
    result = do_execute_command(pipeline, instance, config, mode, tags, solid_selection, preset)
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/dagster/cli/pipeline.py", line 565, in do_execute_command
    return execute_pipeline(
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/dagster/core/execution/api.py", line 373, in execute_pipeline
    return _logged_execute_pipeline(
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/dagster/core/telemetry.py", line 84, in wrap
    result = f(*args, **kwargs)
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/dagster/core/execution/api.py", line 415, in _logged_execute_pipeline
    pipeline_run = instance.create_run_for_pipeline(
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/dagster/core/instance/__init__.py", line 750, in create_run_for_pipeline
    execution_plan = create_execution_plan(
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/dagster/core/execution/api.py", line 743, in create_execution_plan
    resolved_run_config = ResolvedRunConfig.build(pipeline_def, run_config, mode=mode)
  File "/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11/lib/python3.9/site-packages/dagster/core/system_config/objects.py", line 155, in build
    raise DagsterInvalidConfigError(
dagster.core.errors.DagsterInvalidConfigError: Error in config for job
    Error 1: Received unexpected config entry "dask" at path root:execution. Expected: "{ config: { cluster: { existing: { address: (String | { env: String }) } kube(/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11) semexp@datarmor3:/home1/datawork/semexp/workspace/dagit/semaphore-dagster-config/pipelines>     Error 2: Missing required config entry "config" at path root:execution. Sample config for missing entry: {'config': {'cluster': '<selector>'}}
without the 
dask
executor and the 
dask :
in the config, it's better but i need dask :(
2 Views
#dagster-supportJoin Slack
Powered by