https://dagster.io/ logo
Title
d

Daniel Kim

05/24/2021, 2:08 AM
Hello! In my quest to set
base_dir
in a dagstermill example when producing local artifacts on a Windows 10 machine, I found out that if I specify a 
run_coordinator
 in my 
dagster.yaml
 file, then the 
base_dir
 seems to default to where I launch the 
dagster-daemon run
 command. If I don't have 
run_coordinator
 specified in the 
dagster.yaml
 file, then the
base_dir
defaults to where the 
dagit
 command was executed from. Is this expected? I am on version 0.11.10. I know I can set
base_dir
in
local_artifact_storage
specification in the
dagster.yaml
file to override the default location, but per my GH issue (https://github.com/dagster-io/dagster/issues/3898) it is not working for me, presumably due to Windows path.
t

Thomas

05/24/2021, 9:55 AM
Hello, do you have any error logs ?
For info, a real Windows Path (batch) is
C:\Documents\Foo
when Unix is
/home/Foo
You need sometimes to escape special characters
C:\\Documents\\Foo
So I would try this:
local_artifact_storage:
  module: dagster.core.storage.root
  class: LocalArtifactStorage
  config:
    base_dir: "D:\\outputs"
at the end it could be sanitize at the code level if it works
d

daniel

05/24/2021, 1:57 PM
Hi Daniel - not sure about the override issue, but the default location is likely because if you're using the run queue your run is launched from the daemon rather than from dagit. This seems confusing though and I wonder if the DAGSTER_HOME folder would be a better default for both cases (cc @prha)
d

Daniel Kim

05/24/2021, 5:50 PM
@Thomas I do not get error messages. What happens is, in the dagit UI log, it says my notebook output was saved in
.\some_hash.ipynb
Instead of saying
D:\outputs\some_hash.ipynb
I've also tried doubling-up on the back-slash (
D:\\outputs
) as you've suggested in the
dagster.yaml
file, that still did not work. The only thing that has worked for me is using @prha’s suggestion of implementing the
local_file_manager.configured()
method:
"file_manager": local_file_manager.configured(
                    {"base_dir": "D:\outputs"}
                )
👍 1
@daniel For me at least, I do find that confusing. I am just trying to get the iris-kmeans.ipynb example working on my end, but I am not using the same folder structure as the official tutorial. I structure my stuff such that solids are in
solids
folder, pipelines in
pipelines
folder etc. , whereas the tutorial has it all under the same folder. So that is why I am finding out that where the
base_dir
is set is very important to be cognizant of as I am getting the
file not found
error when the notebook is trying to read in the
iris.data
csv file and it is because of my different configuration on my end. After I realize this, I am getting the example to work after I realize that the location of the default
base_dir
can be impacted based on my run configuration. prha's workaround for Windows path is working for me, so I think I am good for now!