https://dagster.io/ logo
#dagster-feedback
Title
# dagster-feedback
b

Brian Pohl

05/11/2023, 9:23 PM
Hello team! I would love if we could push build args into the Docker build process during a Dagster hybrid deployment. This would look like a new input to this workflow, which would then get passed into this step. Why? Our product and development pipeline uses a lot of environment variables, and I’ve had to make many workarounds to access those variables. Here’s my current process: • I spin up the Kubernetes Helm release using Pulumi (basically, Terraform). In my Pulumi script, I pass environment variables received from our CI/CD pipeline to the Helm release. • Our Dagster jobs/steps must access these variables. Because they aren’t available during the Docker build, I can’t reference any environment variables unless the code is inside an op (anything outside an op, i.e. job definition, would be run at build time, and therefore wouldn’t access the variables). • But ops are run by the image spun up by the Helm release, so i can use
os.environ
as long as the code is inside an op. The series of hoops I’ve jumped through work, but I’ve finally run into a scenario where this doesn’t cut it. I want to use an environment variable to construct a string, which gets set in my job config (it’s the dbt target). If I didn’t need to do any string transformations, I could set
resources: dbt: config: target: env: MY_ENV_VARIABLE
, but because I need to modify it, I am forced to do this in my Python scripts. And my Python scripts can only access environment variables during op runtime… So, if I had build args during the build, I could turn all my ARGs into ENVs in the Dockerfile. I could cut Pulumi out of this whole process - passing variables from CI/CD straight into Dagster - and simplify a lot of my Python that today is restricted by only having access to environment variables at runtime.
s

Sean Lopp

05/11/2023, 9:55 PM
We are launching a new hybrid cicd workflow this week that lets you use your own
docker/build-push-action
. The docs are in review but here is a working example: https://github.com/dagster-io/hooli-data-eng-pipelines/blob/master/.github/workflows/deploy-dagster-cloud.yml Having full control of the
build-push-action
should let you build with whatever env vars you want in your image You can deploy this workflow in a branch as well. Note this workflow replaces two yaml files with one since it combines branch and prod deploys into the same workflow.
b

Brian Pohl

05/11/2023, 9:58 PM
this is perfect! how do you distinguish between a prod and a branch deployment then with this workflow?
s

Shalabh Chaturvedi

05/11/2023, 10:01 PM
the
ci-init
step runs the
dagster-cloud ci init
command which inspects the github context. if it is running in a PR (specifically, if HEAD_REF is present) then it determines the branch deployment name by using the dagster cloud API. this is recorded in the build session and used for all subsequent steps.
More specifically, we read the json file at
GITHUB_EVENT_PATH
and look for the
pull_request
key. this is only present for pull requests. docs.
🌈 1
b

Brian Pohl

05/11/2023, 11:26 PM
that’s beautiful. is this officially launched, or should i wait before adopting this?
s

Shalabh Chaturvedi

05/11/2023, 11:29 PM
you can start using it as it is now stable. sometime next week after it is fully official you can replace all the
@v0.1.27
in the example above with
@v0.1
so you track any future fixes. let us know if you run into any issues.
b

Brian Pohl

05/11/2023, 11:30 PM
great, thank you!! 🙏