When I have one physical machine showing up as a docker-based agent in dagster cloud and I want to use it for two separate deployments (i.e. different teams/role-based permissions) and I have environment variables which have the same names but different values - how can I use deployment specific configuration values here? The

env_vars

section seems to be global for the agent and not working per deployment.

I recently had a longer and not fully completed discussion around dynamic partitions in #dagster-support. In particular the idea is based around state derived from a recent asset materialization. Think about I.em a timestamp like last update date or so. If i for whatever reason will need to back load the data I would need to delete an old precious materialization. How can I achieve this in cloud?

Hi! I adapted cicd update-only action to use it as a step of a bitbucket pipeline. I thought it might be helpful for someone, so the demo repo is here. We use GCP to host our dagster instance, so there is also an example of building a user code docker image and pushing it to a GCP registry. Happy hacking! D

When using one physical machine with two. deployed docker runners where each runner points to a dfiferent dagster-cloud deployment I get DSN resolution. failed errors when I want to point the runners to different network segments

Not sure if this a cloud specific question, but thought I'd ask here. We ran a job, and a single op failed due to some underlying data issues. We corrected the issues, and I tried to re-run that op in isolation, but it creates a different run and fails to locate the upstream dependencies for that op, so it won't succeed. This behavior seems different than dagit run locally, where I don't recall ever having dependency issues re-running failed ops in isolation.

hey, i’m attempting to trigger a job run inside a Hex app and then poll for success. i used this code block, passing the appropriate variables, and confirmed that it works locally correctly. i pointed to our stage environment, but i’m running into auth problems. i passed a user-token and still get auth denial.

import os
import requests

TOKEN = os.getenv("DAGSTER_USER_TOKEN")
company = "my-company"
env = "prod"
url = f"https://{company}.dagster.cloud/{env}/graphql"
headers = {"Dagster-Cloud-Api-Token": f"{TOKEN}"}
body_json = {"query": "", "variables: ""}  # omitted for shortness

response = <http://requests.post|requests.post>(url, json=body_json, headers=headers)
response.raise_for_status()

# updated with fix

any thoughts on what might be the issue?

Is there a way to set up capture of Python logging as described here within Dagster Cloud?

just caught up on the community meeting, and very intrigued by the announcement about moving to usage-based pricing for Cloud. Is there a pricing guide available yet?

I'm using the docker cloud agent to launch/sync workspaces and am having trouble adding a code location image that pulls an image from a private dockerhub repo. How do I authenticate the agent with dockerhub?

Hi! Is there any information on GDPR / CCPA compliance? There is no information on this on the privacy policy page.

hey friends, wanted to share a quick note about some awesome work landed by our teammate @daniel who landed this feature, dropped his 🎤 and went off on paternity leave. User code servers now support a configurable time-to-live (TTL). The agent will spin down any user code servers that haven’t served requests recently and will spin them back up the next time they’re needed. You can use this TTL to save compute cost because your user code servers will spend less time sitting idle. You can configure TTL in your agent’s dagster.yaml:

user_code_launcher:
    config:
        server_ttl:
            enabled: true
            ttl_seconds: 7200 # 2 hours

<!channel>, should have also included a quick note introducing myself. For those of you who haven’t already met me, I recently joined the Dagster team as Head of Product woohoo homer in the last month. I previously spent about 4 years at Atlassian focused on Bitbucket Cloud. I’m pumped to be here, and to help serve as a conduit for your feedback and thinking into our feature roadmap. Please don’t hesitate to reach out to me here on this channel, or to DM me anytime 🙂

Got an error in the dagster cloud UI when loading the Execution Time graph for an op:

Operation name: JobMetadataQuery

Message: Internal Server Error (Trace ID: 7566522664540813052)

Path: ["pipelineRunsOrError","results",3,"assets"]

Locations: [{"line":76,"column":3}]

I'm trying to set up the dagster-cloud-ci-cd action so that it can pip install from a separate, private repository. I have a deploy key registered in the separate repository, and I have it mounted in the GH Actions container as SSH key, but it's not clear to me how I can get the docker build step to use the SSH key to use it. Will share current config in 🧵

Hi friends! Happy dagster cloud user here. Just wanted to share some feedback about our experience so far. For context, we're a small team at an early-stage startup, without any dedicated ops folks, so we're all about hosted solutions where we don't have to manage our own infrastructure. We love dagster cloud as it eases some of our infra burden, but with the "hybrid" model we still have to manage the infra that hosts the dagster agent and runs our code. What we'd really love is a fully hosted solution that would take care of provisioning and maintaining all the infrastructure, including the dagster agent. Kind of like heroku for dagster. Is something like that on the long-term roadmap for dagster cloud? Thanks for all your hard work!

Hey friends - we’ve been having a spirited debate about what we should be calling our ‘editor’ role in our RBAC. do you have any preference as to which name fits best for the permission set between ‘viewer’ and ‘admin’?

Which of these names makes the most sense to replace 'editor'

any issues with dagster.cloud right now? can't seem to log in

Hi, I am trying to sign up for dagster cloud via an invite link from email I have tried both firefox and google chrome, when using google sso, it says "authentication failed"

I'm noticing that when a run fails on one repository, if there's a job with the same name in another repository, it will also show up as a failure there

How can I get an alert for a failed sensor? I know that for failed jobs cloud has some specifically improved notification utility

Do you have any plans to support SLA and perhaps send a warning if it was missed?

I have a DBT project which is sourcing its connection details from the environment:

user: "{{ env_var('WAREHOUSE_POSTGRES_USER') }}"

The

dagster.yaml

file of the dagster-cloud runner actually provides these env vars:

run_launcher:
  module: dagster_docker
  class: DockerRunLauncher
  config:
    env_vars:
      - WAREHOUSE_POSTGRES_DB=secret_value
      - WAREHOUSE_POSTGRES_USER=secret_value
      - WAREHOUSE_POSTGRES_PASSWORD=secret_value

However, the pipeline for:

name: myname
on: [pull_request]
jobs:
  preview:
    env:
      WAREHOUSE_POSTGRES_DB: secret_value
      WAREHOUSE_POSTGRES_PASSWORD: secret_value
      WAREHOUSE_POSTGRES_USER: secret_value
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repo
        uses: actions/checkout@v3

      - name: Create Dagster Cloud Code Preview
        uses: dagster-io/dagster-cloud-cicd-action/preview@v0.2.5
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          location-file: ./locations.yaml
          dagit-url: https://${{ secrets.DAGSTER_CLOUD_ORGANIZATION }}.dagster.cloud/${{ secrets.DAGSTER_CLOUD_DEPLOYMENT }}
          api-token: ${{ secrets.DAGSTER_CLOUD_AGENT_TOKEN }}

fails with:

/usr/bin/docker run <http://ghcr.io/my_org/my_project:git_hash|ghcr.io/my_org/my_project:git_hash> dagster-cloud workspace snapshot ***_cloud hash_token --url https://***.dagster.cloud/*** --api-token *** --image <http://ghcr.io/my_org/my_project:git_hash|ghcr.io/my_org/my_project:git_hash> --python-file MWF/repository.py --attribute ***_cloud
  Encountered an error:
  Parsing Error
    Env var required but not provided: 'WAREHOUSE_POSTGRES_USER'

Where do I need to specify the env variables so they can be found when creating the preview? The reason is that dagster-dbt and the CLI there needs these to import the DBT-based assets. It otherwise fails with an errorcode.

I face a problem in one of my partitioned dagster cloud jobs that the partition selector drop-down in the UI seems to be broken in launchpad. Notice - I needed to adapt the partitioning from weekly to daily. Perhaps some DB migration in dagit would be required? In any case currently I must set the tag manually via the edit tag editor. If required, I would be happy to jump on a short call to show you the problem (PM me if desired)

@Dagster Jarred FYI: the preview deploy has one issue though for the Asset Groups tab - when clicking on jobs the link links to jobs inside the preview. Any click on the asset groups moves out of the preview into the real instance.

What's the best Dagster-cloud execution backend for a batch job with 10k tasks that run for a couple minutes each but can almost all be in parallel? I'm trying to figure out which executor will work best for our pipeline. Ideally it would: • Have relatively low per-step overhead, since our tasks don't run for that long (locally, python multiprocess + forkserver is working well). This may not be possible with lots of horizontal scaling / starting more instances. • Have minimal cost while our pipeline isn't running (we would have tasks executing maybe 10 out of 24 hours). I think both EKS and ECS can use Fargate, which will allocate instances/pods to match load. • Scale horizontally to match increasing number of items to process. I think the ECS executor only runs 1 instance per job, not instances per task, which probably rules it out. EKS starts additional pods per step I believe, though that's a bit heavy weight for our needs. A mix of execution models could be nice, not sure if that's at all practical. For example, run the lightweight setup steps with forkserver, and then send 1000 of the heavier tasks each to a separate dynamically allocated instance to be processed there in parallel with forkserver. I'm not familiar with Dask and I don't see Dagster Cloud docs about running an agent for Dask, so I'm not sure if I should be considering it.

Found a massive scheduling issue. I have 3 schedules with the same name and same run time but in different code locations & repos and with different configs. However, it appears that only 1 of them is actually running. I've confirmed that all of them have been turned on

Is there a place in Cloud to view the editor & viewer seat counts for our billing plan? I feel like I've checked every part of the UI but I may have missed something.

I’m trying to experiment with cloud branch deployment and am running into an

UnauthorizedError

I’m manually invoking

dagster-cloud branch-deployment create-or-update

as I’m not using github. I’ve tried with both agent tokens and my user token (I’m an org admin). Any ideas on things to check/try? Right now I’m using a user token configured with

dagster-cloud config setup

and am able to interact with other parts of the cloud environment without error.

Exception: Unable to create or update branch deployment: {'data': {'createOrUpdateBranchDeployment': {'__typename': 'UnauthorizedError'}}}

Trying to log in to cloud, but getting this error about “scheduled maintenance”. When I go to status page all is green.