https://dagster.io/ logo
#ask-community
Title
# ask-community
l

Lawrence Lee

12/19/2022, 6:24 PM
Hi. I want to test Dagster as our workflow orchestrator. We have general experience with Airflow, Prefect, and Flyte. One concept I’m having trouble internalizing is how to “deploy” code into Dagster. I’m reading up on the concepts of Workspaces and Definitions which seem to be different concepts. My use case is as envisioned: I have Dagster deployed in our ECS compute environment. We have a number of users who want to run jobs/ops in ECS. Ideally, they would have their code versioned in a GitHub repo. What is the canonical way to deploy the code to Dagster? If some code has complicated dependencies (e.g. requiring specific libraries or a different Python version than what Dagster is running), can we specify an image/container? TIA! 🙏
dagster bot responded by community 1
m

Mitchell Hynes

12/19/2022, 6:34 PM
Are you using Dagster Cloud or the OSS version? The OSS version you specify a
workspace.yaml
with locations of your code. You probably want multiple gRPC servers with your code in it since you have multiple code locations. Check out this as well For cloud, all it needs is to be able to pull a Docker image from a repository. You can have multiple code-locations to a deployment.
l

Lawrence Lee

12/19/2022, 6:45 PM
OSS version (assuming that is the only way I can install on my ECS infrastructure).
m

Mitchell Hynes

12/19/2022, 6:51 PM
Well, the other way would be to use Hybrid cloud. Basically Dagster manages your code versions, auth, full/branch deployments and other things. All you have to do is run a single agent in ECS and point Dagster at your code locations. The agent pulls those containers from ECR and starts new tasks using the code in them.
The link I gave should get you most of the way there for multiple-code locations using OSS but it’s pretty manual. Hybrid cloud makes code-locations significantly easier if the rest of it fits your use-case (Can only have one deployment, don’t need SAML SSO…).
l

Lawrence Lee

12/19/2022, 7:16 PM
Thanks. From those docs, it seems that if I want to deploy new code, I would have to build a new image with the code and deploy it as a gRPC server.
m

Mitchell Hynes

12/19/2022, 7:18 PM
Yep, then change the
workspace.yaml
to reflect that. BTW: If you want to hot-reload the workspace without restarting Dagit there’s a GraphQL mutation you can use.
Sorry for all the details. I’m currently doing this myself and I’m writing a brain-dump 😆
l

Lawrence Lee

12/20/2022, 5:04 PM
No, the info is very much appreciated and the docs can be a little confusing.