How can I include custom tests & linting in the dagster-cloud branch deployment pipeliens on github? https://github.com/dagster-io/dagster-cloud-hybrid-quickstart/blob/main/.github/workflows/deploy.yml and https://github.com/dagster-io/hooli-data-eng-pipelines/blob/master/.github/workflows/deploy.yml both use dagster_cloud_build_push which is doing many things at once. I want to build the image, test it (tag it) and then push it (assuming the tests are green). There is an example for gitlab: https://github.com/dagster-io/dagster-cloud-action/blob/main/gitlab/hybrid-ci.yml#L113-L124 . there, it would be clear for me how to integrate the testing. But it would be much more convenient if I can use the ready-made dagster github actions plugin but still include testing steps. NOTICE: I am so far still using the old plugin (building the container, testing) and then using the dagster (deprecated old) action to build /push and this is then running into the aforementioned issue that the container is built twice. This is something I want to avoid when re-building the pipeline.

We are actually working on a more modular set of CI actions that separate the image build and the image push, and would allow for an easy "test" step to be added in between I expect these new actions to land next week if you're willing to wait till then

yes - this would be fine

Yes - to elaborate, the new github workflow will have these steps: 1. prerun (validate dagster_cloud.yaml. test authentication to dagster-cloud, create branch deployment if necessary) 2. checkout 3. build (here you will use in your own docker build, test and upload steps) 4.

dagster-cloud ci set-build-output

(update the build session with the image tag from step 3) 5.

dagster-cloud ci deploy

(update dagster-cloud to use the new images) We are also planning to merge the full and branch deployment ymls into a single yml file. Does this sound like something that will work for you?

excellent, i'll ping you next week when this is ready to try out.

@Shalabh Chaturvedi is it already ready?

Hi @geoHeil - these have not landed yet and we're working on resolving some corner cases that have popped up in our testing. Here is a sample of what the new workflow looks like: https://gist.github.com/shalabhc/efba7fa4304dfdd681a3d604484cfa66. This handles both full and branch deployments. If you want to try it out this code should work but is running from a branch

@ci-new-commands

which will need to be replaced later by something like

@v0.1

. If you take a look and have any feedback on the structure, please let us know.

let me see if I find time early next week

Update: if you want to try this out here is a working example of the new workflow: https://github.com/dagster-io/hooli-data-eng-pipelines/blob/master/.github/workflows/deploy-dagster-cloud.yml We will be updating our docs and https://github.com/dagster-io/dagster-cloud-hybrid-quickstart/ in the next few days. Note that: 1. One yaml file implements both branch and full deployments (it detects if it is running in a PR) 2. One job deploys multiple code locations. Previously we had a matrix with one job per code location. The new pattern allows reusing a single docker image for multiple locations, or at least reusing the local docker build cache. The above example deploys two code locations and uses a different docker image for each. 3. The

docker/build-push-action

is directly in your control. You can use any image tag you want as long as you specify it in the

set-build-output

step for each code location.

Where would the tests go? https://github.com/dagster-io/hooli-data-eng-pipelines/blob/master/.github/workflows/deploy-dagster-cloud.yml#L74 seems to already push the image? How can I first build the image, run the tests and then push it if tests succeed and also include layer caching (docker buildx build --cache-from/--cache-to)

The build/push step uses the standard

docker/build-push-action

with

push: true

since that's a common pattern. You can replace it with three steps: build, test, push. An example is in the standard docker github docs here: https://docs.docker.com/build/ci/github-actions/test-before-push/