Hey all, has something changed in dagster serverless deploys? We’re suddenly getting 401 errors. It seems like the issue may be that the graphql api is resolving to: https://dagster.cloud/prod/graphql instead of <https://l.dagster.cloud/prod/graphql%7Chttps://&lt;our_company&gt;.dagster.cloud/prod/graphql> Nothing has changed in our github actions files, and it was working last friday without any issue. full error stack in 🧵

Hi Zach, I'm looking into this. We did release a new version of the workflow yesterday and have seen multiple successful pex deploys with it so it might be something specific to your setup. Can you please share the "Build and deploy Python Executable" section from your branch_deployments.yml?

One other question - can you also share what (if any) environment variables are set at the top of the yml file? Something like this:

env:
  DAGSTER_CLOUD_URL: "<http://myorg.dagster.cloud>"
  DAGSTER_CLOUD_API_TOKEN: ${{ secrets.DAGSTER_CLOUD_API_TOKEN }}
  ENABLE_FAST_DEPLOYS: 'true'

env:
  DAGSTER_CLOUD_URL: ${{ secrets.DAGSTER_CLOUD_URL }}
  DAGSTER_CLOUD_API_TOKEN: ${{ secrets.DAGSTER_CLOUD_API_TOKEN }}
  ENABLE_FAST_DEPLOYS: "true"

and

- name: Build and deploy Python executable
        if: env.ENABLE_FAST_DEPLOYS == 'true'
        uses: dagster-io/dagster-cloud-action/actions/build_deploy_python_executable@pex-v0.1
        with:
          dagster_cloud_file: "$GITHUB_WORKSPACE/project-repo/dagster_cloud.yaml"
          build_output_dir: "$GITHUB_WORKSPACE/build"
          python_version: "3.8"
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Typically our installation replaces this line

DAGSTER_CLOUD_URL: ${{ secrets.DAGSTER_CLOUD_URL }}

with the actual URL, eg

DAGSTER_CLOUD_URL:"<http://myorg.dagster.cloud>"

It is unusual that this is still pointing to the secret. Is it possible the secret value was changed? I don't think secrets can be inspected in github so could you change that to the literal value for your org?

Does

http

vs

https

make any difference here?

Let's use

http

just to be consistent with the pattern we use.

(It'll still communicate over https though in case that's a concern - the server only accepts secure connections)

I guess as context, I was asking because after responding with some of the yaml comments I updated `DAGSTER_CLOUD_URL`just in case it may be wrong. Prior to ~5 minutes ago, it hadn’t been updated in months.

Can you also please share the 'checkout' step above the 'Build and deploy...' section?

Here’s the whole default (fast) deploy job:

dagster_cloud_default_deploy:
    name: Dagster Serverless Deploy
    runs-on: ubuntu-20.04
    outputs:
      build_info: ${{ steps.parse-workspace.outputs.build_info }}

    steps:
      # According to dagster, we were hitting rate limits on deploying new artifacts.
      - name: Workflow Queue
        uses: ahmadnassri/action-workflow-queue@v1.1.0

      - name: Parse cloud workspace
        if: env.ENABLE_FAST_DEPLOYS != 'true'
        id: parse-workspace
        uses: dagster-io/dagster-cloud-action/actions/utils/parse_workspace@v0.1
        with:
          dagster_cloud_file: dagster_cloud.yaml

      - name: Checkout
        if: env.ENABLE_FAST_DEPLOYS == 'true'
        uses: actions/checkout@v3
        with:
          ref: ${{ github.head_ref }}
          path: project-repo

      - name: Build and deploy Python executable
        if: env.ENABLE_FAST_DEPLOYS == 'true'
        uses: dagster-io/dagster-cloud-action/actions/build_deploy_python_executable@pex-v0.1
        with:
          dagster_cloud_file: "$GITHUB_WORKSPACE/project-repo/dagster_cloud.yaml"
          build_output_dir: "$GITHUB_WORKSPACE/build"
          python_version: "3.8"
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Is this running for a pull request update? Can you also verify the very top matches this:

on:
  pull_request:
    types: [opened, synchronize, reopened, closed]

I was wondering about the "company" in the logs as well, that is definitely suspicious - I think your theory is good

yep:

name: Serverless Branch Deployments
on:
  pull_request:
    types: [opened, synchronize, reopened, closed]

Oh haha, nevermind then

the error indicates that the workflow is not able to get information from the checked out repo by running

git -C {project_dir} log -1 --format=%cd --date=unix

. Right now I am unsure why because you are checking out the

head_ref

which should checkout the latest commit in that branch.

My coworker just added another commit to this branch (changes didn’t touch any cicd/github actions stuff) but looks like it triggered a pex file rebuild and passed?

while unlikely, could it be a corrupted git object?

Perhaps? I’m not sure. Alternatively there is a chance our DAGSTER_CLOUD_URL was set to something like

<http://dagster.cloud/company>

instead of

<http://company.dagster.cloud>

, but since it was working as of friday, I’m not sure if this is possible — unsure of what changes were in the recent update.

the offending object is

c5719b8a088d5fdd0fc2c27cc91ea23ec6e360d5

. you could try

git show c5719b8a088d5fdd0fc2c27cc91ea23ec6e360d5

in your local branch to inspect. I think it should be the second last commit in the branch.

Alternatively there is a chance our DAGSTER_CLOUD_URL was set to something like

<http://dagster.cloud/company>

instead of

<http://company.dagster.cloud>

ah interesting. the latest release was mainly refactoring. previously we had more github specific custom code and now we have rolled that into the

dagster-cloud

command. this does change how the

DAGSTER_CLOUD_URL

is passed through. i'll look into if we previously supported

<http://dagster.cloud/company>

style urls.

looks like the git show is working w/o issue.

Hi Zach, wanted to close the loop here: yes it looks like we did break

<http://dagster.cloud/company>

style urls with the earlier release - sorry about that. The fix was rolled out early last week. However I suggest we keep the new style

DAGSTER_CLOUD_URL

in the deploy.yml files.

Thanks for the follow up! We'll keep the suggested format dagsir