https://dagster.io/ logo
j

John Helewa

06/26/2020, 12:47 AM
Is there a good docker-compose container that could be used for testing/learning under windows? I am new to dagster and want to test the basics. The celery-test that Cris put together looks good, but I'm not running on AWS yet, just locally on Windows to try a few things out.
c

Cris

06/26/2020, 12:53 AM
You mean the one I did? it runs local with docker compose. It still needs access to s3 configured if youre using a distributed executor. But you can change that
to use local only
j

John Helewa

06/26/2020, 12:55 AM
Is that an easy configuration setting?
a

alex

06/26/2020, 2:40 PM
so in the
run_config
of each pipeline run you can configure how data is passed from solid to solid. In the
celery
executor you need something remote all the workers can access so you have something like:
Copy code
executor:
  celery: ...
storage:
  s3:
To try something lighter you would do
Copy code
executor:
  multiprocess:
storage:
  filesystem:
which wont use celery for execution but instead just a new child process for each step
j

John Helewa

06/26/2020, 5:12 PM
Thanks Alex, I'll give those a try.