Hello guys ! We are trying to deploy dagster (da...
# ask-community
Hello guys ! We are trying to deploy dagster (dagit + grpc server) using your provided Helm Charts, and as we have multiple dagit deployments to manage, we wanted to use the
option (is there any other way?) That’s why we need to understand how to run grpc server. The project’s structure is quite common I think :
Copy code
├── README.md
├── batchAPIs.py
├── batchPipeline
│   ├── Dockerfile.prod
│   ├── README.md
│   ├── batchPipeline
│   │   ├── __init__.py
│   │   ├── __pycache__
│   │   ├── assets.py
│   │   ├── resources.py
│   │   ├── run_pipeline.py
│   │   └── testDynamicAssets.py
│   ├── batchPipeline_tests
│   │   ├── __init__.py
│   │   └── test_assets.py
│   ├── pyproject.toml
│   ├── schedules
│   │   └── schedules.db
│   ├── setup.cfg
│   ├── setup.py
│   └── workspace.yaml
├── ...
└── requirements.txt
And to launch a user deployment using your helm, as I understood, we need to launch
dagster grpc api <parameters>
do you have an idea on the parameters I need to specify to launch the server ? To launch with
dagster dev
I only need to do :
Copy code
cd batchPipeline
dagit # or dagster dev
NB: This question is related to this earlier topic.
you specify a particular code location to run for the particular grpc server: https://docs.dagster.io/concepts/code-locations/workspace-files#running-your-own-grpc-server - and then in your
you actually point at that running grpc server, rather than specify just the workspace
Thank you for your answer ! But so if I get it right, if we create multiple projects with the cmd
dagster project scaffold --name my-dagster-project
, if we have one main dagster UI running and we want to add the other projects to this UI we have to use grpc servers that will point to only one python file, so let's say
? I must be missing something because in this case the whole project scaffold seems useless
the project structure would become one overarching dagster instance and multiple constituent python packages- each grpc server would point at a particular package, not necessarily a single file
But how can a grpc server point at a particular package ? I might be missing something here
Like if I have my definitions in my
file and my assets in
for example
❤️ 1
A few things: 1. Dagster is running grpc servers behind the scenes for every code location already, pointed at whatever you point your code location at 2. From the docs , for the case where you want to maintain your own grpc server
it’ll slurp up the definitions object at the top level of the packge
(or repo)
Oh it can do that ? Nice !
blob thumbs up 1
So in this case, if we run:
Copy code
dagster api grpc --python-file my-dagster-project/my-dagster-project/assets.py --host --port 4266
It's still gonna load definitions in init.py ?
check out the section on pointing at a python module
The thing is that I don't understand how to use module in Dagster, so the problem must be here
Same problem here, I don’t understand what a python module for dagster should look like
The example are always "my-module", I haven't found a concrete example applicable 😕
You’re waiting here for the name of my asset in my assets.py file ?
Have yall read https://docs.dagster.io/guides/dagster/recommended-project-structure#structuring-your-dagster-project? Outlines a python module structure for a dagster project - and definitions should essentially live at the top of that.
you would expect to have maybe one of these per code location, and then at the top level, either a workspace file, docker-compose or what have you pointing at your gprc servers
@chris We finally understood how to use it and managed to deploy it, thank you for your help !