I am in the process of evaluating dagster cloud. Having an existing (docker-runner via docker-compose-based) setup I am trying to migrate to the managed edition of dagster cloud. However, I do have some questions regarding https://docs.dagster.cloud/guides/continuous-integration My setup is similar to: https://github.com/geoHeil/dagster-ssh-demo/blob/master/docker-compose.yml i.e. a docker-compose-based one with the Queued Run launcher. The desired final result is something similar to: 1. Testing pipeline for each commit: - checkout code - check compliance - if non compliant auto-format and push formatted edition back to github - reformatting, linting, type checking - unit tests - pushing a preview to dagit via dagster cloud 2. Manual review of the deploy preview in dagster cloud. - manual review - when clicking on merge/merging the MR to master continue with (3) 3. Main/Master pipeline - only run on the main branch - all the tests are run again - the version number is incremented - at least via semver - ideally via something like zest release where it is derived from a semantic changelog (https://github.com/zestsoftware/zest.releaser) - the docker image is deployed to the registry - dagster is updating the images and deploying the new version of the code Ideally, I do not need to build the container twice - rather can build it once (i.e. create my conda environment with all the dependencies also only once) and forward / increment the version number and tag it when all the tests have passed successfully.

Hi Georg - just checking something first, do you have a Dagster Cloud account set up? Right now we're in early access so we'd need to work with you to set up a trial if you want to actually see your code in Dagster Cloud once you have the CI/CD set up. For the manual review step you mentioned, you might find this feature helpful: https://docs.dagster.cloud/guides/code-previews - that lets you view a read-only preview of your jobs in Dagster Cloud. The docker registry issue usually points to some kind of missing secret in the Github action so that it can log in to the Docker registry. Would probably need to see the exact details of your github action yaml in order to give more specific guidance on how to proceed

yes that one is set up and the docker-based agent is provisioned

Yeah, the locations.yaml is where you can configure python_file and attribute. There's also a container_context field in locations.yaml which is where things like volumes will go.

I do have one set up alredy - and the agent is running already

ah sorry, I missed that

The additional environment variables would also go there like in dagster.yaml right?

that's right - i just omitted them since in your OSS one they are all postgres related

great.

You can also provide your own Dockerfile and build it however you like

ok

That might be https://github.com/dagster-io/dagster-cloud-cicd-action/blob/main/example/docker-target-stage/locations.yaml

One question whilst the image is building: is it possible to also push the image to ghcr? I see in the logs that the image is created but not pushed

this is the deploy action? That is supposed to run docker build, then docker push

no was preview

oh, preview doesn't currently pus

This means however, that I need to build the conda environment 2x once for the unit tests and then a 2nd time then the dagster-cloud preview task takes over (after the tests were all green)

having a version of the preview action that doesn't build is a reasonable request, I can file an issue for that

you mean straight from the conda env?

i was thinking you would build it, then use it for the preview, then keep that cached build for the deploy

hm as written above ideally yes. But so far I have not really figured out how to do this. Keeping would mean to push it out to the registry though I guess?

Probably, yeah

do you have an example how to enable caching for ghcr?

I don't have a caching example at my fingertips unfortunately

no problem perhaps you could add one in the examples section in the coming weeks?

do you have a full stack trace?

• preview failed 4 minutes ago in 4s 1s 1s 0s Run dagster-io/dagster-cloud-cicd-action/preview@v0.2.4 with: github-token: * location-file: ./locations.yaml dagit-url: https://***.dagster.cloud/*** api-token: * parallel: true Error: Cannot read property 'head' of undefined

you'll need to be logged in using the right docker credentials in the docker context that your agent is using

Also the deploy action of dagster cloud indeed is pushing the image to GHCR - but it is not tagging it with the current tag of the git repository / version from setup.py.

Re the "head" issue - I think the preview action may currently assume that it's being run from a pull request

I was running preview from a PR

that's strange - I'm pretty sure the line that's failing is this:

const commitSha = github.context.payload.pull_request.head.sha;

Also regarding the login:

docker pull <http://ghcr.io/complexity-science-hub/migration-world-formula:337bd4|ghcr.io/complexity-science-hub/migration-world-formula:337bd4>

works fine (locally) but not from the agent.

You may need to add one more line when restarting the agent depending on your environment:

--volume ~/.docker:/root/.docker \

I think this was the issue great one thing is fixed

Can you elaborate on 2)? It usually uses the git hash as the image hash I believe

indeed it does use the hash. But I would prefer if it would use the tag

Which tag exactly?

sure here you go:

name: 2Previews
on: [push]
jobs:
  called_testing: 
    # performs linting, type checking, formatting, ...
    uses: ./.github/workflows/1testing.yml
  preview:
    runs-on: ubuntu-latest
    needs:
      - called_testing
    steps:
      - name: Checkout repo
        uses: actions/checkout@v3

      - name: Create Dagster Cloud Code Preview
        uses: dagster-io/dagster-cloud-cicd-action/preview@v0.2.4
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          location-file: ./locations.yaml
          dagit-url: https://${{ secrets.DAGSTER_CLOUD_ORGANIZATION }}.dagster.cloud/${{ secrets.DAGSTER_CLOUD_DEPLOYMENT }}
          api-token: ${{ secrets.DAGSTER_CLOUD_AGENT_TOKEN }}

If you change it to on [pull_request] I think it may still trigger when you push to a branch that has a pull request active?

indeed - this seems to work

On the tagging topic: The update-only part of the github action is a pretty thin layer on top of a 'dagster-cloud add-location' CLI call described here: https://docs.dagster.cloud/guides/adding-code#adding-a-location I think if you want custom tagging behavior it could make sense for you to make that CLI call directly within your CI/CD - since what you're describing here seems 100% reasonable, but not something that I would necessarily expect users to want in general