https://dagster.io/ logo
#announcements
Title
# announcements
b

Basil V

02/11/2021, 6:58 PM
Hello, I'm trying to run my Dagster setup inside Docker. I'm following this example: https://docs.dagster.io/examples/deploy_docker I downloaded this example and ran
docker-compose up
but something is not working. I'm not getting any errors however the application is not available at localhost:3000. Just to confirm, should this example be runnable as-is or is any other configuration required? Given that I'm not getting any errors, I'm thinking the problem may be with exposing port 3000. I also ran
lsof -i:3000
locally and it shows nothing, making me think the container port isn't being exposed properly. The docker compose file does include the line
expose: - "3000"
in the dagit container though, so not sure what is wrong. Anybody have any suggestions? Thanks so much!
Update: I opened my browser to
http://{virtual-machine ip}:3000
and it is running properly here. So the issue is the port not being exposed. If anybody has an idea what the problem may be please let me know. Thanks!
Update: some googling suggested I needed to add the following to the dagit container:
Copy code
stdin_open: true
tty: true
However, this did not resolve the issue. Still working on it...
The compose file has:
Copy code
expose:
      - "3000"
ports:
      - "3000:3000"
m

max

02/11/2021, 8:34 PM
can you try removing the
expose
chunk
b

Basil V

02/11/2021, 8:34 PM
Yeah let me try that
m

max

02/11/2021, 8:34 PM
@sashank
b

Basil V

02/11/2021, 8:36 PM
Thanks for taking a look at this. My intuition is that 3000 does need to be exposed though either in the dockerfile itself or in the compose file
Yeah it doesn't work with I remove the
expose
chunk
m

max

02/12/2021, 5:09 AM
@daniel any ideas?
d

daniel

02/12/2021, 5:48 AM
I can confirm that the example is supposed to work out of the box, I just verified on my machine that http://localhost:3000/ works for me to load dagit after running 'docker-compose up' on the example. So there must be something different about your Docker environment - have you been able to connect to localhost using other containers in the past?
When I run lsof -i:3000 I'm able to see the port listening
I’d be curious if something like the example here that runs a flask server on port 5000 is also not working, or if it’s something specific to dagster: https://docs.docker.com/compose/gettingstarted/
b

Basil V

02/16/2021, 11:37 PM
Hey @daniel thanks for all this info. So now when I run the lsof command I do see the port mappings, but still not able to access via localhost. I also tried the example flask server and I'm not able to access it on port 5000, however, both for this flask server and for the dagit application, if I access via the <container ip>:3000 or :5000, respectively, I'm able to access the applications. So it must be something with my system config. If you have any guesses as to the problem it is much appreciated, otherwise I'll just keep tinkering. I don't think it is an issue with Dagster though. Thanks!
Another Docker/Dagster question—right now I'm writing files to S3 in my pipeline using dagster-aws. However, within Docker it can't find my credentials. I copied my
~/.aws
file (containing creds/config) into the Docker containers at
/root
but Dagster/boto3 still isn't able to find the credentials. Do you know how to resolve this? (note for actual deployment we plan to use IAM but for testing Docker locally I need to figure out how to pass the creds in via compose). Thanks again!
d

daniel

02/16/2021, 11:39 PM
Not sure if this helps, but I actually ran into a kind of similar issue this afternoon, and it turned out that the server needed to be running on 0.0.0.0 in Docker (not 127.0.0.1) in order to be forwarded correctly. That said, 0.0.0.0 is what the example uses, so I suspect there’s something else going on with your system :/
👍 1
b

Basil V

02/16/2021, 11:41 PM
Thanks! Yeah I'll just have to keep playing with the config. If you have any thought on the S3 AWS config issue please let me know! Thanks so much for the help!
d

daniel

02/16/2021, 11:42 PM
For your newer issue - probably worth a new post in #general, I’m not sure but others on the team with more AWS expertise may be able to help
b

Basil V

02/16/2021, 11:42 PM
Cool thanks!
Hey @daniel I figured out my issue thought I would share. My VirtualBox VM that I created with
docker-machine create ...
was not port forwarding. Likely due to how I installed docker (with Homebrew rather than Docker Desktop for Mac—brew doesn't seem to port forward by default whereas I think Docker Desktop does). Anyways, solution was to go into the VirtualBox VM network adapter and manually add the port forwarding ports. tl;dr if anyone runs into this kind of issue in the future, find out how they installed Docker. If they used brew they may need to manually forward ports. Thanks again for your help!
d

daniel

02/18/2021, 4:54 PM
Oh great! Thanks for the follow-up, glad you're up and running
👍 1
2 Views