@daniel Hi Daniel, thanks for your Github reply regarding gRPC. I just realized that I completely misunderstood the gRPC server for a pipeline code distribution mechanism. In my all-in-one docker example I didn't even realize that the pipeline runs on the gRPC instead of the worker 😑

Hi! No problem - I was actually just looking to see if this user-contributed executor might be closer to what you want: https://github.com/dagster-io/dagster/blob/fd720475ee72c366f2990bb60c4b43205a8575fe/python_modules/libraries/dagster-celery-docker/dagster_celery_docker/executor.py#L72 That would potentially let celery workers execute individual steps, but the actual execution would still be on the gRPC servers, so probably not what you're looking for.

No that won't fit either. Due to the hardware requirements on the workers we're forced to run specific solids on specific workers. How do others typically share code between multiple workers running on different systems?

I see. @alex or @cat are probably better positioned to answer that, I know we have partners using gRPC for code versioning but not sure how they share code across different machines.

How do others typically share code between multiple workers running on different systems?

Typically by deploying the same docker image to the different workers

Packing the workflows up into a Docker image is possible, but not suitable for the quick iterations in the test env. So if we can find a solution that is suitable for testing, Docker is fine for production. We'll probably end up with a smaller env for testing, but one that still has different worker machines with different capabilities. The use case I'm talking about will primarily involve pipeline development: A developer makes repeated changes to one or more solids and pipelines and needs to be able to run and verify the result on a test cluster.

would the volume mount work in the test env?

That would mean shared network mounts. I have my doubt that this will work reliably with many machines/processes accessing the files and a developer trying to update them at the same time... But if you tell me that a lot of people are happy with such a solution, we'll give it a try :-)

yeah it really depends on how the test cluster / environments are managed - but certainly plenty of ways it could go wrong

I thought about git, too, and this is why I was excited about the gRPC server as (git-backed) code server. Although I misunderstood that, as you can read above. Git would be a good solution, if there was a way to automatically reference a git commit when running pipelines. Workers would then be able to grab the pipeline sources from that commit, like this there would be no race conditions when rapidly pushing to the same branch. Something along the lines of: • Developer pushes a new commit containing a pipeline change • Dagit somehow automatically gets notified and reloads the pipeline • The user triggers a pipeline run • The pipeline run is started, passing along the git repo and commit • The workers fetch the commit from the repo and run the pipeline

alright thanks. While we don’t have this now, it was something we had in mind as we built several of the abstractions so something we are considering adding

That sounds good. I'll give the two approaches (shared drive vs git-based) more thought and discuss with the team. Thanks a lot for your inputs!

thank you for sharing details about your constraints 😄