19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Channels
announcements
dagster-airbyte
dagster-airflow
dagster-bigquery
dagster-cloud
dagster-cube
dagster-dask
dagster-dbt
dagster-de
dagster-ecs
dagster-feedback
dagster-kubernetes
dagster-machine-learning
dagster-noteable
dagster-releases
dagster-serverless
dagster-showcase
dagster-snowflake
dagster-spatial
dagster-support
dagster-wandb
dagstereo
data-platform-design
events
faq-read-me-before-posting
gigs-freelance
github-discussions
introductions
jobs
random
tools
豆瓣酱帮
Title
a
Andrea
02/04/2023, 10:12 AMI want to setup dagster for my wide user group. What's the process to deploy a central dagster for many users to deploy jobs/assets from their own environment? Can't find a clear answer in the docs
g
Grigorii Kushnir
02/04/2023, 6:42 PMI think you should take a look at:
https://docs.dagster.io/deployment/overview
and
https://docs.dagster.io/concepts/code-locations
a
Andrea
02/05/2023, 8:01 AMYes I read it. But I still think it's not clear. The code locations example only shows using
dagster devA different documentation page shows a very different approach using a workspace file. This method also does not look scalable. My understanding is that i have projects running with different environments they should be in different code locations. So I would have hundreds of code locations. For each of these I need to spin up a dedicated grpc server which takes resources. Every time I need to make a change or add a code location, I need to change the dagster workspace file itself and potentially reload dagster
The process would be so bad that I'm convinced I don't fully understand it. How is it possible that there is no API to submit a new dag to dagster, no api to manage code locations?
Anything more complex than a demo would not work with the process I mention above
g
Grigorii Kushnir
02/05/2023, 8:08 AMI'm not sure you would need to reload Dagster because with k8s you would have rolling deployment which guarantees 100% uptime.
Here ppl asked the same questions as you:
https://github.com/dagster-io/dagster/discussions/10772
a
Andrea
02/05/2023, 8:09 AMI'm not deploying on k8s
g
Grigorii Kushnir
02/05/2023, 8:10 AMFor a scalable deployment as you've mentioned you better use k8s.
a
Andrea
02/05/2023, 8:10 AMI have my own cluster,not k8s. Every enterprise is different
👎 1
Even so, it would not be a good strategy to reload the dagster pod at every change
g
Grigorii Kushnir
02/05/2023, 8:12 AMYes, it would be your responsibility to guarantee uptime and perform rolling deployment.
a
Andrea
02/05/2023, 8:13 AMThe fact that the orchestrator needs to be modified every time we add or remove projects can't be right, I am sure we are missing something
g
Grigorii Kushnir
02/05/2023, 8:13 AMhttps://github.com/dagster-io/dagster/discussions/8972
Take a look here it might also help, there are cases when ppl do not use code locations and do not spin multiple grpc servers but instead just add repositories.
a
Andrea
02/05/2023, 8:16 AMI mean, thanks a lot for taking the time to answer. Really appreciate it. I'm just baffled the process is so bad. We don't allow connecting to repositories directly on prod to avoid pulling in production of unsecured branches. We deploy everything with CICD where every project has its own NFS mount so they don't interfere with each other storage
So in our case we would have to create hundreds to rpc processes and keep them up. As well as changing and restarting the orchestrator pod every time we have a new deployment (who knows what would happen to stuff that it's running)
Looks like dagster is not for us. I'm just baffled they don't have a simple API to add dags to the orchestrator. Both airflow and Prefect have it
g
Grigorii Kushnir
02/05/2023, 8:18 AMI think reading this discussion might help:
https://github.com/dagster-io/dagster/discussions/10772#discussioncomment-4257899
Looks like dagster is not for us. I'm just baffled they don't have a simple API to add dags to the orchestrator. Both airflow and Prefect have itYou could end up just updating your code in a some specific approach, git submodules maybe from what I imagine you might have.
And Dagster will aoutodiscover your repositories.
a
Andrea
02/05/2023, 8:22 AMI read the article but don't get how it would help. So every one of my projects would have a Definition which says what assets, schedules etc. I need to have for that project. I have hundreds of projects not sharing storage each having its definition file. Each project needs to set up its grpc and dagster needs to be updated every time hard coding the IP of each grpc node to find each definition file. Correct?
g
Grigorii Kushnir
02/05/2023, 8:24 AMI read the article but don't get how it would help. So every one of my projects would have a Definition which says what assets, schedules etc. I need to have for that project. I have hundreds of projects not sharing storage each having its definition file. Each project needs to set up its grpc and dagster needs to be updated every time hard coding the IP of each grpc node to find each definition file. Correct?No, read the whole discussion.
On the github I mean
image.png
a
Andrea
02/05/2023, 8:29 AMMy projects have separate storage, so I won't be able to have a mammoth folder with all my different projects I'm afraid
g
Grigorii Kushnir
02/05/2023, 8:51 AMFor whoever reads this whole, you don't need to restart dagster to discover new code locations:
https://docs.dagster.io/deployment/guides/docker#multi-container-docker-deployment
When you add or remove a user code container, you can also remove the corresponding entry from your workspace.yaml file. If this file is mounted on the Dagit and Dagster daemon containers as a volume, you can pick up the changes in Dagit by reloading from the Deployment tab. The Dagster daemon container will automatically pick up the changes by periodically reloading the workspace.yaml file.