Hello team! I would love if we could push build args into the Docker build process during a Dagster hybrid deployment. This would look like a new input to this workflow, which would then get passed into this step. Why? Our product and development pipeline uses a lot of environment variables, and I’ve had to make many workarounds to access those variables. Here’s my current process: • I spin up the Kubernetes Helm release using Pulumi (basically, Terraform). In my Pulumi script, I pass environment variables received from our CI/CD pipeline to the Helm release. • Our Dagster jobs/steps must access these variables. Because they aren’t available during the Docker build, I can’t reference any environment variables unless the code is inside an op (anything outside an op, i.e. job definition, would be run at build time, and therefore wouldn’t access the variables). • But ops are run by the image spun up by the Helm release, so i can use

os.environ

as long as the code is inside an op. The series of hoops I’ve jumped through work, but I’ve finally run into a scenario where this doesn’t cut it. I want to use an environment variable to construct a string, which gets set in my job config (it’s the dbt target). If I didn’t need to do any string transformations, I could set

resources: dbt: config: target: env: MY_ENV_VARIABLE

, but because I need to modify it, I am forced to do this in my Python scripts. And my Python scripts can only access environment variables during op runtime… So, if I had build args during the build, I could turn all my ARGs into ENVs in the Dockerfile. I could cut Pulumi out of this whole process - passing variables from CI/CD straight into Dagster - and simplify a lot of my Python that today is restricted by only having access to environment variables at runtime.

We are launching a new hybrid cicd workflow this week that lets you use your own

docker/build-push-action

. The docs are in review but here is a working example: https://github.com/dagster-io/hooli-data-eng-pipelines/blob/master/.github/workflows/deploy-dagster-cloud.yml Having full control of the

build-push-action

should let you build with whatever env vars you want in your image You can deploy this workflow in a branch as well. Note this workflow replaces two yaml files with one since it combines branch and prod deploys into the same workflow.

this is perfect! how do you distinguish between a prod and a branch deployment then with this workflow?

the

ci-init

step runs the

dagster-cloud ci init

command which inspects the github context. if it is running in a PR (specifically, if HEAD_REF is present) then it determines the branch deployment name by using the dagster cloud API. this is recorded in the build session and used for all subsequent steps.

that’s beautiful. is this officially launched, or should i wait before adopting this?

you can start using it as it is now stable. sometime next week after it is fully official you can replace all the

@v0.1.27

in the example above with

@v0.1

so you track any future fixes. let us know if you run into any issues.

great, thank you!! 🙏