The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

Or should I be using environment variables instead (which these already are for the database config, just had pipeline specific config via yaml)
edit: reading <https://docs.dagster.io/tutorial/advanced_pipelines#parametrizing-pipelines-with-resources> it showed that `resources.yaml` is included
edit2: Ah, it is probably presets!

To add another option - you can have pipeline presets read from Env variables.

Personally I would:

Capture any non sensitive config as presets
Have sensitive config in the presets configured to read from environment variables

See StringSource docs: <https://docs.dagster.io/_apidocs/config|https://docs.dagster.io/_apidocs/config>

Yep! That's exactly what I want to do, I just didn't want to have an environment variable for non-secret values.

Fantastic! This is exactly what I needed, means I can split out my yaml config as required, and the secret values are in env (as defined in the yaml) and the non secret in yaml. Thanks so much :slightly_smiling_face: