https://dagster.io/ logo
Join Slack
Powered by
I want to setup dagster for my wide user group. Wh...
# ask-community
a
I want to setup dagster for my wide user group. What's the process to deploy a central dagster for many users to deploy jobs/assets from their own environment? Can't find a clear answer in the docs
g
a
Yes I read it. But I still think it's not clear. The code locations example only shows using 
dagster dev
as a way to load code locations. But how to add/change code locations on an existing dagster deployment without having to restart it?
A different documentation page shows a very different approach using a workspace file. This method also does not look scalable. My understanding is that i have projects running with different environments they should be in different code locations. So I would have hundreds of code locations. For each of these I need to spin up a dedicated grpc server which takes resources. Every time I need to make a change or add a code location, I need to change the dagster workspace file itself and potentially reload dagster
The process would be so bad that I'm convinced I don't fully understand it. How is it possible that there is no API to submit a new dag to dagster, no api to manage code locations?
Anything more complex than a demo would not work with the process I mention above
g
I'm not sure you would need to reload Dagster because with k8s you would have rolling deployment which guarantees 100% uptime.
Here ppl asked the same questions as you: https://github.com/dagster-io/dagster/discussions/10772
a
I'm not deploying on k8s
g
For a scalable deployment as you've mentioned you better use k8s.
a
I have my own cluster,not k8s. Every enterprise is different
👎 1
Even so, it would not be a good strategy to reload the dagster pod at every change
g
Yes, it would be your responsibility to guarantee uptime and perform rolling deployment.
a
The fact that the orchestrator needs to be modified every time we add or remove projects can't be right, I am sure we are missing something
g
https://github.com/dagster-io/dagster/discussions/8972 Take a look here it might also help, there are cases when ppl do not use code locations and do not spin multiple grpc servers but instead just add repositories.
a
I mean, thanks a lot for taking the time to answer. Really appreciate it. I'm just baffled the process is so bad. We don't allow connecting to repositories directly on prod to avoid pulling in production of unsecured branches. We deploy everything with CICD where every project has its own NFS mount so they don't interfere with each other storage
So in our case we would have to create hundreds to rpc processes and keep them up. As well as changing and restarting the orchestrator pod every time we have a new deployment (who knows what would happen to stuff that it's running)
Looks like dagster is not for us. I'm just baffled they don't have a simple API to add dags to the orchestrator. Both airflow and Prefect have it
g
Looks like dagster is not for us. I'm just baffled they don't have a simple API to add dags to the orchestrator. Both airflow and Prefect have it
You could end up just updating your code in a some specific approach, git submodules maybe from what I imagine you might have.
And Dagster will aoutodiscover your repositories.
a
I read the article but don't get how it would help. So every one of my projects would have a Definition which says what assets, schedules etc. I need to have for that project. I have hundreds of projects not sharing storage each having its definition file. Each project needs to set up its grpc and dagster needs to be updated every time hard coding the IP of each grpc node to find each definition file. Correct?
g
I read the article but don't get how it would help. So every one of my projects would have a Definition which says what assets, schedules etc. I need to have for that project. I have hundreds of projects not sharing storage each having its definition file. Each project needs to set up its grpc and dagster needs to be updated every time hard coding the IP of each grpc node to find each definition file. Correct?
No, read the whole discussion.
On the github I mean
image.png
a
My projects have separate storage, so I won't be able to have a mammoth folder with all my different projects I'm afraid
g
For whoever reads this whole, you don't need to restart dagster to discover new code locations: https://docs.dagster.io/deployment/guides/docker#multi-container-docker-deployment
Copy code
When you add or remove a user code container, you can also remove the corresponding entry from your workspace.yaml file. If this file is mounted on the Dagit and Dagster daemon containers as a volume, you can pick up the changes in Dagit by reloading from the Deployment tab. The Dagster daemon container will automatically pick up the changes by periodically reloading the workspace.yaml file.
3 Views
Open in Slack
PreviousNext
Powered by