https://dagster.io/ logo
Join Slack
Powered by
Hey guys, just wanted to point out that the change...
# announcements
d
Hey guys, just wanted to point out that the changes in https://github.com/dagster-io/dagster/pull/2640 broke my setup. We run dagster via docker compose for hacking locally:
Copy code
dagster:
  image: company/project
  build: .
  command: sh -c "cp pipelines/dagster.yaml tmp/dagster/ && dagit -h 0.0.0.0 -p 9090 -w pipelines/workspace.yaml"
  environment:
    DAGSTER_HOME: tmp/dagster
  ports:
    - 9090:9090
  volumes:
    - ./tmp/dagster:/src/tmp/dagster
Worked fine on 0.7.13, but today I upgraded to 0.8.5 and getting this error:
Copy code
dagster.core.errors.DagsterInvariantViolationError: DAGSTER_HOME must be absolute path: tmp/dagster
Changing the path to be absolute in the docker compose file is a no go, it'll break on another dev's machine. Any non-hacky suggestions?
Going to downgrade to before this change was released for now
Sorry, just to clarify. The compose setup can be fixed by changing 
tmp/dagster
to the absolute path inside the container, which doesn't change between machines. What my brain was trying to say 😃 was that when devs try hacking locally using the same setup as our docker compose as above, but outside of any container, it breaks.
So if I run for example:
Copy code
cd /home/danny/project
cp pipelines/dagster.yaml tmp/dagster
DAGSTER_HOME=/home/danny/project/tmp/dagster dagit -h 0.0.0.0 -p 9090 -w pipelines/workspace.yaml
It'll fail when another dev chose to run out of 
/home/josh/project
What's the rationale for forcing absolute paths?
Ok so https://github.com/dagster-io/dagster/issues/2609#issuecomment-647571145 explains a bit. I guess it makes sense it needs to be absolute if an external system like cron is involved.
a
you should still be able to use 
~
if you want user relative paths
if im reading your setup correctly - shouldn’t $DAGSTER_HOME inside the container consistently be 
/src/tmp/dagster
? (assuming the workdir of the image is 
/src/
d
Yeah, that's why I clarified above, the compose setup is just showing how we've laid out the paths. But when a dev starts dagster directly (outside a container) they'll need to customize any script or docs that set DAGSTER_HOME. Specifically what I wrote here https://dagster.slack.com/archives/CCCR6P2UR/p1593460539338100?thread_ts=1593459690.337300&cid=CCCR6P2UR
And using 
~
won't help because some deployments or devs might have it under 
/opt/project
etc etc
Relative paths were working fine for our use case on 0.7.13. I think it would be ideal to keep them, since that's the default UX a typical user like myself expects and is how most *nix stuff works. It sounds from the PR that using 
DAGSTER_HOME=.
messes things up due to cron being in the picture. I've never dug into the internals so can't speak about a better solution here, except to say that the current 3 line "fix" will probably cause upgrade friction to other users like myself, and I wish there was an alternative that stayed relaxed about the path.
a
alright thanks for the feedback I’ll consider what alternatives we have for preventing the class of issues this defensive check exists to prevent
in the mean time, for the developer facing shell scripts you could do something like
Copy code
SCRIPT=`realpath $0`
SCRIPTPATH=`dirname $SCRIPT`
DAGSTER_HOME="$SCRIPTPATH/tmp/dagster"
to resolve the shell script relative path to absolute before setting the env var
🙏 1
👍 1
Open in Slack
PreviousNext
Powered by