Are there best practices for managing credentials within a dagster pipeline (especially when using a scheduler)? I know Airflow has some support for this built in to the UI that essentially centralizes the credentials in a DB. didn't know if this is addressed and I missed it or if there are better ways?

hey Bradley! we don’t have any built-in support right now. we see many teams using systems like Hashicorp Vault or otherwise having their own existing secrets management systems, and so we haven’t built any secrets management directly into Dagster

That makes sense

do you have an existing system at your company? if not, is there a preferred way you’d like to see this work?

I think we do use something like Vault

I took a stab at building an example for how credentials could work which I think you will find useful! The idea is the following, you use whatever tech you use for credential stuff but eventually you will have those creds stored as environment variables on your machines. If that condition holds, in dagster, you can access those variables the following way: Config view: https://github.com/dagster-io/dagster/blob/master/examples/dagster_examples/bay_bikes/environments/weather_base.yaml#L4 How the cred management resource works: https://github.com/dagster-io/dagster/blob/master/examples/dagster_examples/bay_bikes/resources.py#L75 How to configure the cred management system in to your pipelines: https://github.com/dagster-io/dagster/blob/master/examples/dagster_examples/bay_bikes/pipelines.py#L58 How to use the cred management system in your solids: https://github.com/dagster-io/dagster/blob/master/examples/dagster_examples/bay_bikes/solids.py#L159

Awesome. Will dig through these. Thanks!

Let me know if this works for you, if not I would love to learn more about your use case and factor it into our design when we embark on a house view

Sure thing