Is there a way to use a single GitHub action to sync multipl dagster #dagster-plus

Is there a way to use a single GitHub action to sy...

Charlie Bini

01/09/2023, 10:30 PM

Is there a way to use a single GitHub action to sync multiple code locations, each with their own Dockerfile? I can make it work with multiple locations and one Dockerfile, but I want each code location's image to use a different Dockerfile. Getting an error I can't explain on GitHub when I try

Charlie Bini

01/09/2023, 10:30 PM

I put the different Dockerfiles in their own directories and pointed

dagster-cloud.yaml

to the respective directories for each location, but they all fail with an error along the lines of

Copy code

Error: buildx failed with: ERROR: failed to solve: failed to compute cache key: failed to calculate checksum of ref femc2novkejrbkoun33sp29rh::ljzx0b7tyxk77r80nfm2x6j88: failed to walk /tmp/buildkit-mount2643354055/src/teamster: lstat /tmp/buildkit-mount2643354055/src/teamster: no such file or directory

jordan

01/09/2023, 10:45 PM

Hm. That looks fairly low-level in docker build. It looks to me like the actual

src/teamster

(presumably that’s the name of your project?) isn’t getting copied into the build directory? what does your dagster-cloud.yaml look like and what does your github action’s yaml look like?

Charlie Bini

01/09/2023, 10:46 PM

yeah, project name is teamster and the code in a src folder

Charlie Bini

01/09/2023, 10:47 PM

example location:

Copy code

locations:
  - location_name: kippnewark
    code_source:
      python_file: src/teamster/kippnewark/definitions.py
    build:
      directory: .docker/kippnewark/
      registry: us-central1-docker.pkg.dev/teamster-332318/teamster-kippnewark/teamster-kippnewark
    working_directory: /root/app
    container_context:
      k8s:
        env_secrets:
          - kippnewark

Charlie Bini

01/09/2023, 10:48 PM

action yaml

Copy code

name: Hybrid Branch Deployment

on:
  pull_request:
    branches:
      - main
    types:
      - opened
      - synchronize
      - reopened

concurrency:
  # Cancel in-progress runs on same branch
  group: ${{ github.ref }}
  cancel-in-progress: true

jobs:
  parse_workspace:
    runs-on: ubuntu-latest

    outputs:
      build_info: ${{ steps.parse-workspace.outputs.build_info }}

    steps:
      # <https://github.com/actions/checkout>
      - name: Checkout repo
        uses: actions/checkout@v3

      # <https://github.com/dagster-io/dagster-cloud-action>
      - name: Parse cloud workspace
        id: parse-workspace
        uses: dagster-io/dagster-cloud-action/actions/utils/parse_workspace@v0.1
        with:
          dagster_cloud_file: .dagster/dagster-cloud.yaml

  dagster_cloud_build_push:
    name: Dagster Hybrid Deploy
    runs-on: ubuntu-latest
    needs: parse_workspace

    strategy:
      fail-fast: false
      matrix:
        location: ${{ fromJSON(needs.parse_workspace.outputs.build_info) }}

    permissions:
      contents: read
      id-token: write
      pull-requests: write

    steps:
      # <https://github.com/actions/checkout>
      - name: Checkout
        uses: actions/checkout@v3
        with:
          ref: ${{ github.head_ref }}

      # <https://github.com/google-github-actions/auth>
      - id: auth
        name: Authenticate to Google Cloud
        uses: google-github-actions/auth@v1
        with:
          token_format: access_token
          workload_identity_provider:
            projects/${{ secrets.GCP_PROJECT_NUMBER
            }}/locations/global/workloadIdentityPools/github-pool/providers/github-provider
          service_account:
            ${{ secrets.GCP_SERVICE_ACCOUNT_NAME }}@${{ secrets.GCP_PROJECT_ID
            }}.<http://iam.gserviceaccount.com|iam.gserviceaccount.com>

      # <https://github.com/docker/login-action>
      - name: Login to GAR
        uses: docker/login-action@v2
        with:
          registry: ${{ secrets.GCP_REGION }}-docker.pkg.dev
          username: oauth2accesstoken
          password: ${{ steps.auth.outputs.access_token }}

      # <https://github.com/dagster-io/dagster-cloud-action>
      - name: Build and deploy to Dagster Cloud hybrid
        uses: dagster-io/dagster-cloud-action/actions/hybrid_branch_deploy@v0.1
        with:
          organization_id: ${{ secrets.DAGSTER_ORGANIZATION_ID }}
          dagster_cloud_api_token: ${{ secrets.DAGSTER_CLOUD_AGENT_TOKEN }}
          location: ${{ toJson(matrix.location) }}
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

jordan

01/10/2023, 1:48 AM

You might need to add a step that copies

src/teamster/*

into the

.docker/kippnewark/