Hello All, I am using Dask executor specifying local filesystem storage

"storage": {"filesystem": {"config": {"base_dir": '/dagster'}}}

, in the run_config. However I get surprising error

Undefined field "storage" at the root. Expected:

. Wonder if the storage has been overwritten somewhere? I also set mode as follows local_mode =

ModeDefinition(

name='local',

resource_defs={

's3': s3_resource,

'tempfile': tempfile_resource,

'file_cache': fs_file_cache,

'file_manager': local_file_manager,

},

executor_defs=default_executors + [dask_executor],

intermediate_storage_defs=s3_plus_default_intermediate_storage_defs,

system_storage_defs=s3_plus_default_storage_defs,

logger_defs={'local_logger': json_file_logger}

Dagster version 0.9.9. Thanks for suggestions in advance!

I believe it has to do with setting

intermediate_storage_defs=s3_plus_default_intermediate_storage_defs,

which I believe causes the run config key to change to

intermediate_storage:

cc @sandy

@alex (cc @sandy) - thanks for speedy reply. So is this an expected behaviour and if so is there any workaround short of disabling

intermediate_storage

(it does work if I remove it!). Or should I raise an issue on GitHub? Thanks!

I notice that youre setting both "system_storage_defs" and "intermediate_storage_defs"

@sandy thanks - so I have removed

system_storage_defs

and

storage

and set

'intermediate_storage':{'s3': {'config': {'s3_bucket': 'test'}}}

in run_config but still get the same error, presumably due to https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dask/dagster_dask/executor.py#L143

ahh, so apparently the dask executor isn't compatible with intermediate storage. I'll look at addressing this. in the mean time, best to stick with system_storage_defs

thanks @sandy - will do, could you say if this is a substantial piece of work (i.e. is this a big change)? Just asking as both dask executor and intermediate storage are important functionalities for us?

it's not a big change, and I'm currently looking into the fix - however, currently "system storage" and "intermediate storage" do exactly the same thing, so you shouldn't be losing out on any functionality by configuring system storage instead of intermediate storage. the reason we have both of them around is that we're in the process of deprecating "system storage" for naming reasons