https://dagster.io/ logo
Join Slack
Powered by
Hi guys, I'm facing an issue when i'm using dask e...
# ask-community
t
Hi guys, I'm facing an issue when i'm using dask executor on my job then filling the dask config as follow :
Copy code
execution:
  dask:
    config:
      cluster:
        pbs:
          queue: mpi_4
          project: dagster
          walltime: '00:05:00'
          memory: '120gb'
          cores: 28
          resource_spec: 'select=4:ncpus=28:mem=120gb'
          n_workers: 1
I got this error :
Copy code
raise DagsterInvalidConfigError(
dagster.core.errors.DagsterInvalidConfigError: Error in config for job
    Error 1: Received unexpected config entry "dask" at path root:execution. Expected: "{ config: { cluster: { existing: { address: (String | { env: String }) } kube(/home/datawork-semaphore-exp/conda/conda_envs/semaphore-scripts-1.0.11) semexp@datarmor3:/home1/datawork/semexp/workspace/dagit/semaphore-dagster-config/pipelines>     Error 2: Missing required config entry "config" at path root:execution. Sample config for missing entry: {'config': {'cluster': '<selector>'}}
And my job looks like that :
Copy code
@job(
    name='template_runner_job',
    resource_defs={"io_manager": fs_io_manager},
    executor_def=dask_executor
)
def template_runner_job():
    template_runner_op()
p
Hi Thomas. I think you can drop the 
dask
in your config:
Copy code
execution:
  config:
    cluster:
      existing:
        ...
Are you using the Launchpad in Dagit to compose the config? If not, you may find the autocomplete / config validation feature useful.
t
If i'm using the launchpad to compose the config i can't insert 
dask
because of a control on the interface that doesn't allow me this keyword
In order to pass through this issue, i'm passing my config directly via dagster and i got the error i post below 🙂
If i drop the 
dask
in my config, how can i configure my dask_executor that i put on my job ?
p
cc @chris Thomas, can you share the error that you’re getting?
c
Pretty sure using dask executor with config is broken right now on jobs - https://github.com/dagster-io/dagster/pull/5848 will fix
t
@chris Hi, I pick the change you've made in 
<https://github.com/dagster-io/dagster/pull/5848/files/8efc6be1cc2e6795e56fb002846b9d17ef694720#diff-f4477f5a8ae52e60950dc28d9f884f39f9912e850e02692dce9b0fb325ae280b|dagster_dask/executor.py>
Here is what i get on the interface
I can't hit the launch button if 
dask
is in my config
@chris @prha And via dagster, with a command like that 
dagster job execute -f myjob.py -c myconfig.yaml
: I got this
Copy code
Error 1: Received unexpected config entry "dask" at path root:execution. Expected: "{ config: { cluster: { existing: { address: (String | { env: String }) } kube?: { } local?: { } lsf?: { } moab?: { } oar?: { } pbs?: { } sge?: { } slurm?: { } ssh?: { } yarn?: { } } } }".
   Error 2: Missing required config entry "config" at path root:execution. Sample config for missing entry: {'config': {'cluster': '<selector>'}}
c
After you've picked the change in that PR - you should not have 
dask
in your config. Since jobs only have one executor, there is no need to additionally specify the executor name in execution config. Since 
executor_name
is no longer part of the schema, you're getting an error. If you get rid of the 
dask
keyword, it should work
2 Views
Open in Slack
PreviousNext
Powered by