Daniel Kim
05/24/2021, 2:08 AMbase_dir
in a dagstermill example when producing local artifacts on a Windows 10 machine, I found out that if I specify a run_coordinator
in my dagster.yaml
file, then the base_dir
seems to default to where I launch the dagster-daemon run
command. If I don't have run_coordinator
specified in the dagster.yaml
file, then the base_dir
defaults to where the dagit
command was executed from. Is this expected? I am on version 0.11.10.
I know I can set base_dir
in local_artifact_storage
specification in the dagster.yaml
file to override the default location, but per my GH issue (https://github.com/dagster-io/dagster/issues/3898) it is not working for me, presumably due to Windows path.Thomas
05/24/2021, 9:55 AMC:\Documents\Foo
when Unix is /home/Foo
You need sometimes to escape special characters
C:\\Documents\\Foo
So I would try this:
local_artifact_storage:
module: dagster.core.storage.root
class: LocalArtifactStorage
config:
base_dir: "D:\\outputs"
at the end it could be sanitize at the code level if it worksdaniel
05/24/2021, 1:57 PMDaniel Kim
05/24/2021, 5:50 PM.\some_hash.ipynb
Instead of saying D:\outputs\some_hash.ipynb
I've also tried doubling-up on the back-slash ( D:\\outputs
) as you've suggested in the dagster.yaml
file, that still did not work. The only thing that has worked for me is using @prha’s suggestion of implementing the local_file_manager.configured()
method:
"file_manager": local_file_manager.configured(
{"base_dir": "D:\outputs"}
)
solids
folder, pipelines in pipelines
folder etc. , whereas the tutorial has it all under the same folder. So that is why I am finding out that where the base_dir
is set is very important to be cognizant of as I am getting the file not found
error when the notebook is trying to read in the iris.data
csv file and it is because of my different configuration on my end. After I realize this, I am getting the example to work after I realize that the location of the default base_dir
can be impacted based on my run configuration. prha's workaround for Windows path is working for me, so I think I am good for now!