Are there best practices for managing credentials ...
# announcements
b
Are there best practices for managing credentials within a dagster pipeline (especially when using a scheduler)? I know Airflow has some support for this built in to the UI that essentially centralizes the credentials in a DB. didn't know if this is addressed and I missed it or if there are better ways?
n
hey Bradley! we don’t have any built-in support right now. we see many teams using systems like Hashicorp Vault or otherwise having their own existing secrets management systems, and so we haven’t built any secrets management directly into Dagster
b
That makes sense
n
do you have an existing system at your company? if not, is there a preferred way you’d like to see this work?
b
I think we do use something like Vault
I was wondering more for personal projects
a
I took a stab at building an example for how credentials could work which I think you will find useful! The idea is the following, you use whatever tech you use for credential stuff but eventually you will have those creds stored as environment variables on your machines. If that condition holds, in dagster, you can access those variables the following way: Config view: https://github.com/dagster-io/dagster/blob/master/examples/dagster_examples/bay_bikes/environments/weather_base.yaml#L4 How the cred management resource works: https://github.com/dagster-io/dagster/blob/master/examples/dagster_examples/bay_bikes/resources.py#L75 How to configure the cred management system in to your pipelines: https://github.com/dagster-io/dagster/blob/master/examples/dagster_examples/bay_bikes/pipelines.py#L58 How to use the cred management system in your solids: https://github.com/dagster-io/dagster/blob/master/examples/dagster_examples/bay_bikes/solids.py#L159
This is by no means perfect or the house view as we are still figuring out the right integrations as Nate mentioned above, but this should give you an abstracted view of how to solve the problem (or at the very least, give you something to work with for a personal project XD)
b
Awesome. Will dig through these. Thanks!
a
Let me know if this works for you, if not I would love to learn more about your use case and factor it into our design when we embark on a house view
b
Sure thing