Responding out of thread since this is asked a lot...
# announcements
n
Responding out of thread since this is asked a lot, the new io managers system allows for this nicely 🙂
👍 3
r
I think I explained myself badly though, what I meant is that now I am mounting a volume in docker compose to the docker_example_pipelines image where I have a "data" folder to store results in a .csv, and I'd like to be able to read this csv from outside docker. The initial container for docker_example_pipelines is reacting properly to my volume, but I cannot manage to get the pipeline run containers spinned by the daemon to write into this "data" folder. Is there any way to have a volume shared with these executors?
n
What launcher mode are you using? dagster_docker presumably?
r
run_launcher: module: dagster_docker class: DockerRunLauncher
n
Yes, from the example it's on that one
Then no
The volume config on the generated container is not configurable
As I said, you would use an IO manager instead of writing to the local filesystem
That might be Minio running in a persistent container or something
But still, same idea
👀 1
r
Cool, I'm just writing the end results there, and I also upload them to GDrive for analysts. The issue is that some data scientists review the results once the run is finished and for them it's easier to just read local files. I guess if that is my use case I then should stick to a single container like here https://docs.dagster.io/deploying/docker and mount a volume to it
Is there any plan in near term to add any kind of support for volumes in the run executors?
Or IOManagers is the way to go here with MinIO or similar
n
I'm not sure how many people are using dagster_docker for large scale, complex stuff
I see it mostly for quick demos 🙂
r
yeah I was using it as a demo before moving to k8s
and wanted to do it similar to what we would do there (ofc no local files there) but for testing purposes
n
Right, and there you would 100% have to use network storage of some kind
r
I wanted to have some local files 🙂
n
Yeah, so just hasn't really been a priority
Since it would be a test of something not possible outside of the affordances of a demo, not a very good test 😉
r
So understandable haha! Thanks for your support though Noah, I'm learning a lot from you guys since I post a lot of questions here, I'm new to Dagster haha