is it possible to use environment variables in dagster.yaml and dagster_cloud.yaml?

Yes, it is possible to use environment variables in both

dagster.yaml

and

dagster_cloud.yaml

files. For

dagster.yaml

, environment variables can be used to parameterize your Dagster instance configuration. However, there are some limitations to be aware of. For example, an issue was reported where environment variables were not being correctly interpreted for non-string types like boolean or int within

dagster.yaml

. This suggests that while environment variables can be used, there may be some constraints or bugs affecting their use in certain contexts. For

dagster_cloud.yaml

, the documentation provides an example of how to set environment variables for specific code locations by adding them to your agent's configuration in the

dagster_cloud.yaml

file. The

container_context

property in this file is used to set the variables in the agent's environment. Here is an example from the documentation:

yaml # dagster_cloud.yaml locations:   - location_name: cloud-examples     image: dagster/dagster-cloud-examples:latest     code_source:       package_name: dagster_cloud_examples     container_context:       docker:         env_vars:           - DATABASE_NAME           - DATABASE_USERNAME=hooli_testing

In this example,

DATABASE_NAME

would be pulled from the local environment, and

DATABASE_USERNAME

is explicitly set to

hooli_testing

. Remember that when setting environment variables for a full deployment in Dagster Cloud, the values will bypass Dagster Cloud entirely as they are pulled from the user cluster. When setting them on a per-code location basis, the values will pass through Dagster Cloud. Please ensure you have the necessary permissions in Dagster Cloud to set environment variables, and refer to the specific documentation for your deployment type (e.g., Amazon ECS, Docker, Kubernetes) for detailed instructions on configuring

container_context

.

@Scout no I meant to use env vars to set the configurations, for example, for

image

, I’d want that value to come from an env var, something like

image: {env.image_name}

In Dagster, you can use environment variables within your YAML configuration files by using a special syntax to reference the environment variables. This allows you to dynamically set configuration values based on the environment in which your Dagster instance or Dagster Cloud agent is running. Here's how you can reference an environment variable in your

dagster.yaml

or `dagster_cloud.yaml`:

yaml image: {"env": "IMAGE_NAME"}

In this example,

IMAGE_NAME

is the name of the environment variable that contains the image name you want to use. When Dagster processes this configuration, it will substitute

{"env": "IMAGE_NAME"}

with the value of the

IMAGE_NAME

environment variable from the environment. This approach is useful for managing secrets and sensitive information, as well as for providing flexibility in your deployment configurations. It allows you to avoid hardcoding values and makes your configuration more portable and secure. Please note that the actual syntax for referencing environment variables may vary depending on the context within the YAML file and the specific Dagster version you are using. Always refer to the latest Dagster documentation for the most accurate and up-to-date information on using environment variables in your configuration files.