Hello All I am just getting started with Dagster a...
# ask-community
u
Hello All I am just getting started with Dagster and managed to stand up the tutorial project. Can someone please help with the following questions: 1) Are there any sample projects on GitHub or in the Dagster docs where Assets are organised into multiple .py files instead of just the one assets.py . Same goes for whatever is defined in init .py 2) Is it possible to integrate Dagster with keycloak, other auth. Are there docs ? Can there be different user roles in Dagster. Ideally we may not want the same user to have access to perform all actions on the UI including maintaining jobs and also any admin activity. 3) What is the recommendation on how big a Dagster project should be? E.g. should all the data pipelines with over 15-20 different jobs each in my business unit be under one Dagster project? Can we make the Dagster environment multitenant? 4) Can assets, ops , schedule i.e. all the python files be deployed to the project using a Dagster API? if not, as part of deployment do we have to just copy the changed .py to the project folder? 5) Is there doc on the differences between Dagster cloud features vs Dagster open source. Is secrets mgmt available using a UI in Dagster open source?
o
Hi Upen - 1. Yes- check this file out https://github.com/dagster-io/dagster/tree/master/examples/project_fully_featured 2. Not sure about keycloak, but we do have different roles in cloud 3. Check out this guide https://docs.dagster.io/guides/dagster/recommended-project-structure, you can have different code locations in open source, or multiple deployments in cloud 4. Was there something you were trying to do with CLI that we can help with? 5. We have audit trail, role based access and branch deployments in cloud but all of the core product is available through open source
u
Thanks @Odette Harary (2) Is there any other way we can integrate an external auth provider, identity provider with open source Dagster? We don't plan to use Dagster cloud yet. (5) I didn't understand the part where you said that all of the core product is available open source. E.g. in the open source dagit UI I don't see a way to enter secrets. What other such features, functionality do we miss if we are not using Dagster cloud? On the other questions I think I'm good for now
o
Hi Upen - yes you can store secrets as environment variables