https://dagster.io/ logo
#announcements
Title
# announcements
s

Sergii Ivakhno

09/29/2020, 1:39 PM
Hello All, I am using Dask executor specifying local filesystem storage
"storage": {"filesystem": {"config": {"base_dir": '/dagster'}}}
, in the run_config. However I get surprising error
Undefined field "storage" at the root. Expected:
. Wonder if the storage has been overwritten somewhere? I also set mode as follows local_mode =
ModeDefinition(
name='local',
resource_defs={
's3': s3_resource,
'tempfile': tempfile_resource,
'file_cache': fs_file_cache,
'file_manager': local_file_manager,
},
executor_defs=default_executors + [dask_executor],
intermediate_storage_defs=s3_plus_default_intermediate_storage_defs,
system_storage_defs=s3_plus_default_storage_defs,
logger_defs={'local_logger': json_file_logger}
Dagster version 0.9.9. Thanks for suggestions in advance!
a

alex

09/29/2020, 2:28 PM
I believe it has to do with setting
intermediate_storage_defs=s3_plus_default_intermediate_storage_defs,
which I believe causes the run config key to change to
intermediate_storage:
cc @sandy
s

Sergii Ivakhno

09/29/2020, 3:10 PM
@alex (cc @sandy) - thanks for speedy reply. So is this an expected behaviour and if so is there any workaround short of disabling
intermediate_storage
(it does work if I remove it!). Or should I raise an issue on GitHub? Thanks!
s

sandy

09/29/2020, 3:11 PM
I notice that youre setting both "system_storage_defs" and "intermediate_storage_defs"
It's preferable to only set intermediate_storage_defs
and then in your config you can have an "intermediate_storage:" section instead of a "storage:" section
s

Sergii Ivakhno

09/29/2020, 3:18 PM
@sandy thanks - so I have removed
system_storage_defs
and
storage
and set
'intermediate_storage':{'s3': {'config': {'s3_bucket': 'test'}}}
in run_config but still get the same error, presumably due to https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dask/dagster_dask/executor.py#L143
s

sandy

09/29/2020, 3:19 PM
ahh, so apparently the dask executor isn't compatible with intermediate storage. I'll look at addressing this. in the mean time, best to stick with system_storage_defs
s

Sergii Ivakhno

09/29/2020, 3:25 PM
thanks @sandy - will do, could you say if this is a substantial piece of work (i.e. is this a big change)? Just asking as both dask executor and intermediate storage are important functionalities for us?
s

sandy

09/29/2020, 3:27 PM
it's not a big change, and I'm currently looking into the fix - however, currently "system storage" and "intermediate storage" do exactly the same thing, so you shouldn't be losing out on any functionality by configuring system storage instead of intermediate storage. the reason we have both of them around is that we're in the process of deprecating "system storage" for naming reasons
👍 1