^ related to <@U042A10Q1CH>’s post above – I think...
# ask-community
k
^ related to @Daniel Daum’s post above – I think it would help out the community enormously if you guys would consider publishing images & improving the docs for Docker usage. It'd be great to have: 1. A canonical Docker Hub image that works with both Dagit and Dagster Daemon 2. A revised
docker-compose.yml
example that parameterizes all config to a
.env
file, and uses volumes to store code & data. a. Container 1 =
dagster-daemon
, uses above image b. Container 2 =
dagster-dagit
, uses above image c. Container 3 =
dagster-executor
, uses whatever image people want to run their code, such as a prebuilt python one from jupyter-stacks, and uses a bind mount to a local code directory rather than copying over a single
repo.py
each time (not very helpful), as well as another bind mount to store results d. Container 4 =
db
– include an example which uses vanilla Postgres – although I suspect many people will have a preexisting one already they can use The current docs are really confusing and ideally there'd be a simple block you could add to an existing
docker-compose.yml
without a ton of fuss.
👍 6
👍🏽 1
🙏 1
c
Thanks for the valuable feedback Kevin--I agree that publishing these images as you mentioned would make it much easier to get started with docker and dagster
t
Yeah! Thanks for pointing this out, Kevin! We can do a lot more with our images. The first one is in the works, and we need to triage the second one. By chance, do you already have your described setup working? If so, I'd would love to help you become a contributor and raise a PR back to Dagster to update our docs and point to the images on a registry.
k
Yep @Tim Castillo I've got it working, how about I recreate a minimal example for you guys to check out and we can go from there
i think the container 3 image might require some iteration because it needs to both install dependencies that people need to run their code, but also have the GRPC server as the entry point
Sure there is a way to do that w/ a prebuild image but I'm not a docker expert 😄
@Tim Castillo above sound good to you?
t
Hey Kevin!! Sorry for the delay on replying. and I appreciate the follow up! Yeah, feel free to create the minimal example in your own time, let me know what you think the last mile things are, and I'll work with you to take it across the finish line.
k
hey @Tim Castillo no worries – here's a skeleton we can work from https://github.com/kevinschaich/dagster-docker-example/tree/master two problems right now:
❤️ 1
1. I haven't managed to get it working with Postgres if I use a base image other than a vanilla one like
python:3.7-slim
, because of the issues mentioned here https://github.com/dagster-io/dagster/issues/11353
2. I haven't managed to get it working with
DockerRunLauncher
, and the example here https://github.com/dagster-io/dagster/tree/1.1.20/examples/deploy_docker uses the old
@repository
decorator which I think we should change to use the new
Definitions
api
hey @Tim Castillo @daniel got both those issues fixed now – I can have an updated example up in a few hours. there's a lot of thorny pieces to get Dagster, Postgres, and Spark all happily working together – I think it'd be valuable to have a reproducible example of that somewhere. would you mind if we start with a more full-featured example including an asset which uses Spark & a custom I/O manager, and then if you feel it's too complicated, we can also create a more minimal one?
what would be the best way to review together? should I make a pull request to dagster w/ the contents of this repo overwriting the existing
examples/deploy_docker
folder?