https://dagster.io/ logo
#dagster-cloud
Title
# dagster-cloud
g

geoHeil

04/28/2023, 4:29 PM
How can I include custom tests & linting in the dagster-cloud branch deployment pipeliens on github? https://github.com/dagster-io/dagster-cloud-hybrid-quickstart/blob/main/.github/workflows/deploy.yml and https://github.com/dagster-io/hooli-data-eng-pipelines/blob/master/.github/workflows/deploy.yml both use dagster_cloud_build_push which is doing many things at once. I want to build the image, test it (tag it) and then push it (assuming the tests are green). There is an example for gitlab: https://github.com/dagster-io/dagster-cloud-action/blob/main/gitlab/hybrid-ci.yml#L113-L124 . there, it would be clear for me how to integrate the testing. But it would be much more convenient if I can use the ready-made dagster github actions plugin but still include testing steps. NOTICE: I am so far still using the old plugin (building the container, testing) and then using the dagster (deprecated old) action to build /push and this is then running into the aforementioned issue that the container is built twice. This is something I want to avoid when re-building the pipeline.
s

Sean Lopp

04/28/2023, 4:58 PM
We are actually working on a more modular set of CI actions that separate the image build and the image push, and would allow for an easy "test" step to be added in between I expect these new actions to land next week if you're willing to wait till then
g

geoHeil

04/28/2023, 5:10 PM
yes - this would be fine
please ping me when they are ready
s

Shalabh Chaturvedi

04/28/2023, 5:12 PM
Yes - to elaborate, the new github workflow will have these steps: 1. prerun (validate dagster_cloud.yaml. test authentication to dagster-cloud, create branch deployment if necessary) 2. checkout 3. build (here you will use in your own docker build, test and upload steps) 4.
dagster-cloud ci set-build-output
(update the build session with the image tag from step 3) 5.
dagster-cloud ci deploy
(update dagster-cloud to use the new images) We are also planning to merge the full and branch deployment ymls into a single yml file. Does this sound like something that will work for you?
g

geoHeil

04/28/2023, 5:14 PM
dagster punk
yes
s

Shalabh Chaturvedi

04/28/2023, 5:15 PM
excellent, i'll ping you next week when this is ready to try out.
🌈 1
g

geoHeil

05/05/2023, 7:00 AM
@Shalabh Chaturvedi is it already ready?
s

Shalabh Chaturvedi

05/05/2023, 2:19 PM
Hi @geoHeil - these have not landed yet and we're working on resolving some corner cases that have popped up in our testing. Here is a sample of what the new workflow looks like: https://gist.github.com/shalabhc/efba7fa4304dfdd681a3d604484cfa66. This handles both full and branch deployments. If you want to try it out this code should work but is running from a branch
@ci-new-commands
which will need to be replaced later by something like
@v0.1
. If you take a look and have any feedback on the structure, please let us know.
g

geoHeil

05/05/2023, 2:39 PM
let me see if I find time early next week
s

Shalabh Chaturvedi

05/11/2023, 12:51 AM
Update: if you want to try this out here is a working example of the new workflow: https://github.com/dagster-io/hooli-data-eng-pipelines/blob/master/.github/workflows/deploy-dagster-cloud.yml We will be updating our docs and https://github.com/dagster-io/dagster-cloud-hybrid-quickstart/ in the next few days. Note that: 1. One yaml file implements both branch and full deployments (it detects if it is running in a PR) 2. One job deploys multiple code locations. Previously we had a matrix with one job per code location. The new pattern allows reusing a single docker image for multiple locations, or at least reusing the local docker build cache. The above example deploys two code locations and uses a different docker image for each. 3. The
docker/build-push-action
is directly in your control. You can use any image tag you want as long as you specify it in the
set-build-output
step for each code location.
g

geoHeil

05/12/2023, 8:25 PM
Where would the tests go? https://github.com/dagster-io/hooli-data-eng-pipelines/blob/master/.github/workflows/deploy-dagster-cloud.yml#L74 seems to already push the image? How can I first build the image, run the tests and then push it if tests succeed and also include layer caching (docker buildx build --cache-from/--cache-to)
s

Shalabh Chaturvedi

05/12/2023, 8:47 PM
The build/push step uses the standard
docker/build-push-action
with
push: true
since that's a common pattern. You can replace it with three steps: build, test, push. An example is in the standard docker github docs here: https://docs.docker.com/build/ci/github-actions/test-before-push/
For layer caching, you would add the
cache-to
and
cache-from
to the same action. example from these docs:
Copy code
name: Build and push
        uses: docker/build-push-action@v4
        with:
          context: .
          push: true
          tags: user/app:latest
          cache-from: type=gha
          cache-to: type=gha,mode=max
This is a good point - we will add the
cache-*
parameters in our example since it is usually recommended.
2 Views