you should be able to reset your state by uninstalling the app from your GH account

you can do that from here https://github.com/settings/installations

i think it’s a bug if it’s confusing. there are a lot of states users can be in during the GH auth process. we tried to test every one but some fail more elegantly than others (for reasons outside of our control sometimes)

I did revoke the access and uninstall the app in Github. How do I set it up again now? I’m getting GraphQL errors on the cloud site, but it looks like its still showing the first couple set-up steps (including Github integration) as complete.

I went through the connection process to Github this morning, but afterwards, back on the set-up page, I still see the “Choose account or organization” button. If I click it again, I see the org I set up, with an option to “Cancel Request”. Is there a workflow in Github or internally that this might be waiting on.

Well. Now the next step in deployment is available - to clone the starter project. But when I specify a repo name and click on “Clone and Deploy”, I get an “Internal Server Error” message. Also, trying to use “Configure Installation” on my Github connection gets me a 404 error.

I was able to get to a page in a round-about way at: https://github.com/apps/dagster-cloud/installations/29145624. But that shows only permissions on selected repositories (with none selected). I selected all repositories in the setup, and the link that is being pointed to in the UI (where I get the 404) is different than above: https://github.com/apps/dagster-cloud/installations/28843402.

@Pete Hunt or @ben Should I be opening a new thread for this? Was trying to keep everything together. Upshot is we have an install now with “All repository” permissions, but the ID we see in our URL is 29145624. The one in the “Configure” URL on the Dagster cloud site had 28843402.

The URL from the cloud site doesn’t exist in our GH org. I don’t know how that got crossed up.

This seems to be causing the issue that we can’t create the starter project in GH.

On the GH side, I see the app, and I see it marked for “All repositories.” Is there anything else I should see out there?

It looks like this might be a permissions issue; something we probably need to patch on our end. Do you happen to know if your user account has permissions to create a repo in the org?

For context; some GH API interactions can only be accomplished when tied to a user, while others occur using the GH app installation without a user attached to the action. I think what may be happening is that the repository creation is being tied to your user account which may not have perms to create a repo in the org. We should be performing the action without a user affiliated instead.

Yes, I was just able to create a repo using that account.

Could two-factor auth be causing an issue? I do have an access token associated with that account if there’s a way to set that up in the cloud.,

but it sounds like the service user does have that permission, at least when interacting through the UI?

Very sorry you’re running into this, it’s a strange issue & one we haven’t seen before.

Hmmm…I just tried to fork that repo over to our account, and I got an error message:

You cannot fork this repository to the selected destination due to a policy.

Could this be what’s interfering with your code, too?

but the code that’s failing is the repo creation itself; not the subsequent push it looks like

still, I wonder if it’s the same policy that’s restricting automation creating the repo

Hey @ben, are there any needs for more elevated permissions than normal repo-interaction activities (pulls, pushes, PRs), once we get past the initial repo creation? I’m thinking of seeing if I can get extra permissions added for this first step on our side, then tamp them back down. If that doesn’t work, I can try to clone the repo instead.

So it pretty much has all permissions on our side.

Just the “Connect to Github” button.

Any org admin should see the same things, right? So it’s odd that I see the connection that was set up and that account doesn’t.

If it’s the account in the connection that matters, then everyone should see the same thing…

I’m curious if the manual “use this template” flow will work now that the user has elevated permissions, or if there is still an error there

We could potentially get it working manually, but there’s enough strangeness going on here to not be confident about other potential issues going forward. It’d be great to get to the bottom of this.

I have availability after 2 PM PST today, 10 AM - 1 PM PST tomorrow, 8-9 AM or after 9:30 AM PST Thursday.

Is 12:00 PST, 3:00 EST tomorrow OK? You can send me an invite at kevin.martin@theguarantors.com.

Hey @ben, thanks for helping us to get up and running. Now I move on to struggling through how everything connects together. The first thing I’m having issues with is how to pass secrets. I have a test (fake) secret defined in Github, and I’m trying to extend the cereals example code to get that secret value into an environment variable and use it within one of the sample asset methods just to write it out to a context log. I don’t see where I put a run_config - all the examples I find so far in the docs pass the run config as part of the execute_in_process() method to run inline. But the example code loads the assets into a repository using load_assets_from_package_module() and the actual run is invoked online. I think I understand the part of adding config_schema to the asset decorator, but where does the code for the run_config go that passes in the config value? Or am I thinking about this completely wrong?

Thanks. What about getting that value from an environment variable that was passed in from GH secrets?

Where do I register the secrets in the code? I added a “SUPER_SECRET” key in Github and documented it in both the env: sections in each of the deployment and branch_deployment gihub workflow config files, and added the configuration in the Materialize UI as you pointed out. But I’m getting an error saying that there is no such environment variable set.

ops:

nabisco_cereals:

config:

super_secret:

env: SUPER_SECRET

dagster._core.errors.DagsterInvalidConfigError: Error in config for job

Error 1: Post processing at path root:ops:nabisco_cereals:config:super_secret of original value {'env': 'SUPER_SECRET'} failed:

dagster._config.errors.PostProcessingError: You have attempted to fetch the environment variable "SUPER_SECRET" which is not set. In order for this execution to succeed it must be set in this environment.

Stack Trace:

File "/usr/local/lib/python3.8/site-packages/dagster/_config/post_process.py", line 79, in _post_process

new_value = context.config_type.post_process(config_value)

File "/usr/local/lib/python3.8/site-packages/dagster/_config/source.py", line 42, in post_process

return str(_ensure_env_variable(cfg))

File "/usr/local/lib/python3.8/site-packages/dagster/_config/source.py", line 16, in _ensure_env_variable

raise PostProcessingError(

I think this may be because the changes to the deployment workflows I made were in the branch. I assume the workflow files actually pulled for the GH actions are the ones in main. Trying out some changes now.

And here’s the main deploy script:

name: Serverless Prod Deployment
on:
  push:
    branches:
      - "main"
      - "master"
concurrency:
  # Cancel in-progress deploys to main branch
  group: ${{ github.ref }}
  cancel-in-progress: true
env:
  DAGSTER_CLOUD_URL: "<http://dagster.cloud/theguarantors>"
  DAGSTER_CLOUD_API_TOKEN: ${{ secrets.DAGSTER_CLOUD_API_TOKEN }}
#  SUPER_SECRET: ${{ secrets.SUPER_SECRET }}

jobs:
  parse_workspace:
    runs-on: ubuntu-latest
    outputs:
      build_info: ${{ steps.parse-workspace.outputs.build_info }}
    steps:
      - uses: actions/checkout@v3
      - name: Parse cloud workspace
        id: parse-workspace
        uses: dagster-io/dagster-cloud-action/actions/utils/parse_workspace@v0.1
        with:
          dagster_cloud_file: dagster_cloud.yaml

  dagster_cloud_build_push:
    runs-on: ubuntu-latest
    needs: parse_workspace
    name: Dagster Serverless Deploy
    strategy:
      fail-fast: false
      matrix:
        location: ${{ fromJSON(needs.parse_workspace.outputs.build_info) }}
    steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          ref: ${{ github.head_ref }}
      - name: Build and deploy to Dagster Cloud serverless
        uses: dagster-io/dagster-cloud-action/actions/serverless_prod_deploy@v0.1
        with:
          dagster_cloud_api_token: ${{ secrets.DAGSTER_CLOUD_API_TOKEN }}
          location: ${{ toJson(matrix.location) }}
          # Uncomment to pass through Github Action secrets as a JSON string of key-value pairs
          # env_vars: ${{ toJson(secrets) }}
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          #  SUPER_SECRET: ${{ secrets.SUPER_SECRET }}

OK. That works now. Thanks!