What are the environment variables which are set o...
# dagster-plus
What are the environment variables which are set on branch deployments? From https://docs.dagster.io/guides/dagster/branch_deployments#testing-against-production-with-dagster-cloud-branch-deployments I can see
Copy code
When building custom IO managers - let`s say for ML flor or file based data (HFS, S3, ....) I might want to include the branch into the asset key prefix for branch deployments. How can I obtain a reference to the current branch? For such a branching IO manager where can I find examples? https://github.com/dagster-io/hooli-data-eng-pipelines/blob/master/hooli_data_eng/definitions.py#L108-L190 seems to only switch to
and then supply different credentials (the same for all branches). I want to have something different: Similar to DBT sim-ci a different logical key prefix (branch name) should be added somewhere (and yes most likely not the production resource should be used as well.
@Sean Lopp this page is giving me a 404
can you fix the 404 and then share an example of how these are used in IO managers?
I believe the links should now be working: https://docs.dagster.io/guides/dagster/branch_deployments#testing-against-production-with-dagster-cloud-branch-deployments Please let me know if this is not the case for you Georg
Great. This works
Do you also have an example for how to do this in other non-snowflake IO managers? Do you have best practices if the branch deployment should be handled at the top level of dagster configuration (resource level items) or inside the IO managers where an filesystem IO manager might prefix PR-Ids of an S3 bucket?
I think the most common pattern is to do the creation of the resource either in a job or IO manager and the configuration in the top-level resource For example, if you had files on s3 and wanted to put them in a PR folder, I’d configure the folder prefix as a top level resource config and then if necessary build that folder in the IO manager. This way the new resources UI coming in 1.3 (available as an experimental user feature flag setting now) will show the config value which is helpful for knowing where things are being written to during dev and staging There are more examples here https://github.com/dagster-io/hooli-data-eng-pipelines/blob/master/hooli_data_eng/definitions.py#L82