https://dagster.io/ logo
#dagster-cloud
Title
# dagster-cloud
z

Zach P

03/01/2023, 6:31 PM
Hey all, has something changed in dagster serverless deploys? We’re suddenly getting 401 errors. It seems like the issue may be that the graphql api is resolving to: https://dagster.cloud/prod/graphql instead of <https://l.dagster.cloud/prod/graphql%7Chttps://&lt;our_company&gt;.dagster.cloud/prod/graphql> Nothing has changed in our github actions files, and it was working last friday without any issue. full error stack in 🧵
👀 3
Error snippet
s

Shalabh Chaturvedi

03/01/2023, 6:41 PM
Hi Zach, I'm looking into this. We did release a new version of the workflow yesterday and have seen multiple successful pex deploys with it so it might be something specific to your setup. Can you please share the "Build and deploy Python Executable" section from your branch_deployments.yml?
d

daniel

03/01/2023, 6:42 PM
One other question - can you also share what (if any) environment variables are set at the top of the yml file? Something like this:
Copy code
env:
  DAGSTER_CLOUD_URL: "<http://myorg.dagster.cloud>"
  DAGSTER_CLOUD_API_TOKEN: ${{ secrets.DAGSTER_CLOUD_API_TOKEN }}
  ENABLE_FAST_DEPLOYS: 'true'
z

Zach P

03/01/2023, 6:42 PM
Copy code
env:
  DAGSTER_CLOUD_URL: ${{ secrets.DAGSTER_CLOUD_URL }}
  DAGSTER_CLOUD_API_TOKEN: ${{ secrets.DAGSTER_CLOUD_API_TOKEN }}
  ENABLE_FAST_DEPLOYS: "true"
and
Copy code
- name: Build and deploy Python executable
        if: env.ENABLE_FAST_DEPLOYS == 'true'
        uses: dagster-io/dagster-cloud-action/actions/build_deploy_python_executable@pex-v0.1
        with:
          dagster_cloud_file: "$GITHUB_WORKSPACE/project-repo/dagster_cloud.yaml"
          build_output_dir: "$GITHUB_WORKSPACE/build"
          python_version: "3.8"
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
s

Shalabh Chaturvedi

03/01/2023, 6:51 PM
Typically our installation replaces this line
Copy code
DAGSTER_CLOUD_URL: ${{ secrets.DAGSTER_CLOUD_URL }}
with the actual URL, eg
Copy code
DAGSTER_CLOUD_URL:"<http://myorg.dagster.cloud>"
It is unusual that this is still pointing to the secret. Is it possible the secret value was changed? I don't think secrets can be inspected in github so could you change that to the literal value for your org?
z

Zach P

03/01/2023, 6:52 PM
Does
http
vs
https
make any difference here?
s

Shalabh Chaturvedi

03/01/2023, 6:54 PM
Let's use
http
just to be consistent with the pattern we use.
d

daniel

03/01/2023, 6:55 PM
(It'll still communicate over https though in case that's a concern - the server only accepts secure connections)
z

Zach P

03/01/2023, 6:57 PM
I guess as context, I was asking because after responding with some of the yaml comments I updated `DAGSTER_CLOUD_URL`just in case it may be wrong. Prior to ~5 minutes ago, it hadn’t been updated in months.
When I updated it with https I get more errors, but different ones — going to retry with http just in case to see if that makes any difference.
Okay, after updating got new errors — the same if it’s set to http vs https (I wonder if previously the secret may have been set to http://dagster.cloud/company ?)
errors_updated_dagster_cloud_url
s

Shalabh Chaturvedi

03/01/2023, 7:05 PM
Can you also please share the 'checkout' step above the 'Build and deploy...' section?
z

Zach P

03/01/2023, 7:07 PM
Here’s the whole default (fast) deploy job:
Copy code
dagster_cloud_default_deploy:
    name: Dagster Serverless Deploy
    runs-on: ubuntu-20.04
    outputs:
      build_info: ${{ steps.parse-workspace.outputs.build_info }}

    steps:
      # According to dagster, we were hitting rate limits on deploying new artifacts.
      - name: Workflow Queue
        uses: ahmadnassri/action-workflow-queue@v1.1.0

      - name: Parse cloud workspace
        if: env.ENABLE_FAST_DEPLOYS != 'true'
        id: parse-workspace
        uses: dagster-io/dagster-cloud-action/actions/utils/parse_workspace@v0.1
        with:
          dagster_cloud_file: dagster_cloud.yaml

      - name: Checkout
        if: env.ENABLE_FAST_DEPLOYS == 'true'
        uses: actions/checkout@v3
        with:
          ref: ${{ github.head_ref }}
          path: project-repo

      - name: Build and deploy Python executable
        if: env.ENABLE_FAST_DEPLOYS == 'true'
        uses: dagster-io/dagster-cloud-action/actions/build_deploy_python_executable@pex-v0.1
        with:
          dagster_cloud_file: "$GITHUB_WORKSPACE/project-repo/dagster_cloud.yaml"
          build_output_dir: "$GITHUB_WORKSPACE/build"
          python_version: "3.8"
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
s

Shalabh Chaturvedi

03/01/2023, 7:12 PM
Is this running for a pull request update? Can you also verify the very top matches this:
Copy code
on:
  pull_request:
    types: [opened, synchronize, reopened, closed]
d

daniel

03/01/2023, 7:13 PM
I was wondering about the "company" in the logs as well, that is definitely suspicious - I think your theory is good
z

Zach P

03/01/2023, 7:13 PM
yep:
Copy code
name: Serverless Branch Deployments
on:
  pull_request:
    types: [opened, synchronize, reopened, closed]
FWIW
company
is me search and replacing the name of our company with
company
😛
d

daniel

03/01/2023, 7:14 PM
Oh haha, nevermind then
s

Shalabh Chaturvedi

03/01/2023, 7:19 PM
the error indicates that the workflow is not able to get information from the checked out repo by running
git -C {project_dir} log -1 --format=%cd --date=unix
. Right now I am unsure why because you are checking out the
head_ref
which should checkout the latest commit in that branch.
z

Zach P

03/01/2023, 7:22 PM
My coworker just added another commit to this branch (changes didn’t touch any cicd/github actions stuff) but looks like it triggered a pex file rebuild and passed?
s

Shalabh Chaturvedi

03/01/2023, 7:23 PM
while unlikely, could it be a corrupted git object?
z

Zach P

03/01/2023, 7:25 PM
Perhaps? I’m not sure. Alternatively there is a chance our DAGSTER_CLOUD_URL was set to something like
<http://dagster.cloud/company>
instead of
<http://company.dagster.cloud>
, but since it was working as of friday, I’m not sure if this is possible — unsure of what changes were in the recent update.
s

Shalabh Chaturvedi

03/01/2023, 7:31 PM
the offending object is
c5719b8a088d5fdd0fc2c27cc91ea23ec6e360d5
. you could try
git show c5719b8a088d5fdd0fc2c27cc91ea23ec6e360d5
in your local branch to inspect. I think it should be the second last commit in the branch.
Alternatively there is a chance our DAGSTER_CLOUD_URL was set to something like
<http://dagster.cloud/company>
instead of
<http://company.dagster.cloud>
ah interesting. the latest release was mainly refactoring. previously we had more github specific custom code and now we have rolled that into the
dagster-cloud
command. this does change how the
DAGSTER_CLOUD_URL
is passed through. i'll look into if we previously supported
<http://dagster.cloud/company>
style urls.
👀 1
z

Zach P

03/01/2023, 7:35 PM
looks like the git show is working w/o issue.
👍 1
s

Shalabh Chaturvedi

03/16/2023, 7:08 PM
Hi Zach, wanted to close the loop here: yes it looks like we did break
<http://dagster.cloud/company>
style urls with the earlier release - sorry about that. The fix was rolled out early last week. However I suggest we keep the new style
DAGSTER_CLOUD_URL
in the deploy.yml files.
z

Zach P

03/16/2023, 8:44 PM
Thanks for the follow up! We'll keep the suggested format dagsir
2 Views