If I’ve got some other system that needs to tell dagster cloud to run something, can I do so through some kind of API?

I have some runs that have been running for a few days with a somewhat significant amount of logs that are no longer able to load in Dagit (they were able to load consistently in 10-20s prior to the scheduled maintenance today). I'm getting this graphQL error in Dagit when I try to load them:

Operation name: RunRootQuery

Message: Internal Server Error (Trace ID: 6611625006052030400)

Path: ["pipelineRunOrError","assets"]

Locations: [{"line":24,"column":3}]

If I remember correctly these runs had somewhere around 7 or 8k event logs and maybe 1 or 2k debug/info/warning/error logs. Somewhat similarly a job I have for scraping Dagster run info from the graphQL API started failing with a gateway timeout error this afternoon when querying 10 runIds at a time via

runsOrError

(I can adjust this to query less runs at a time, but this was having no issues until around when the maintenance window hit)

when i hover over the

materialize

button, i am told that i can

shift + click

to add config, but even if i don't, i am prompted for config. is that expected?

The usage report on the job-level usage throws a graphql exception for me and downloads an empty CSV file.

I don't have a Youtube acct, so asking here. Do you have any plans to bring branch deployment functionality to open source?

Hey, got very enthusiastic about Cloud after having joined Dagster Day dagster spin Would it be possible to schedule a call with one of you to ask some specific question about Dagster Cloud before we’d start our trial? Or would you prefer written communication?

Hello, I got this error. I'm running a k8s agent. What does this mean?

Multiprocess executor: child process for step asset_overview_table.parse_asset_overview_xml[globalexusafull_LGDF_20220729_004171_065_FUL_35086254_xml] unexpectedly exited with code -9
dagster._core.executor.child_process_executor.ChildProcessCrashException

Stack Trace:
  File "/usr/local/lib/python3.9/site-packages/dagster/_core/executor/multiprocess.py", line 210, in execute
    event_or_none = next(step_iter)
,  File "/usr/local/lib/python3.9/site-packages/dagster/_core/executor/multiprocess.py", line 324, in execute_step_out_of_process
    for ret in execute_child_process_command(multiproc_ctx, command):
,  File "/usr/local/lib/python3.9/site-packages/dagster/_core/executor/child_process_executor.py", line 163, in execute_child_process_command
    raise ChildProcessCrashException(exit_code=process.exitcode)

how does one view usage reports in Dagster Cloud?

Hi I'm trying to use a custom dockerfile in the cicd, and I think it might not work

Github actions gives the following error when committing to the main branch. Anything obvious that I am missing?

GraphQLStorageError: Invalid URL '***/prod/graphql': No scheme supplied. 
Perhaps you meant http://***/prod/graphql?
Could not load registry information  - your serverless deployment may still be activating.

trying to migrate to the new GH workflow, and it's failing with this error

Error when evaluating 'strategy' for job 'dagster_cloud_build_push'. .github/workflows/dev-deploy.yaml (Line: 48, Col: 19): Error parsing fromJson,.github/workflows/dev-deploy.yaml (Line: 48, Col: 19): Error reading JToken from JsonReader. Path '', line 0, position 0.,.github/workflows/dev-deploy.yaml (Line: 48, Col: 19): Unexpected type of value '', expected type: Sequence.

it's a big day today. I started keeping a dagster cloud tab open on my phone!

ok now I'm trying to do a branch deployment, but getting this error during

Run dagster-io/dagster-cloud-action/actions/utils/notify@main

Traceback (most recent call last):
  File "/create_or_update_comment.py", line 86, in <module>
    main()
  File "/create_or_update_comment.py", line 82, in main
    pr.create_issue_comment(message)
  File "/usr/local/lib/python3.8/site-packages/github/PullRequest.py", line 456, in create_issue_comment
    headers, data = self._requester.requestJsonAndCheck(
  File "/usr/local/lib/python3.8/site-packages/github/Requester.py", line 353, in requestJsonAndCheck
    return self.__check(
  File "/usr/local/lib/python3.8/site-packages/github/Requester.py", line 378, in __check
    raise self.__createException(status, responseHeaders, output)
github.GithubException.GithubException: 403 {"message": "Resource not accessible by integration", "documentation_url": "<https://docs.github.com/rest/reference/issues#create-an-issue-comment>"}

When performing a preview deployment with a changed/updated repository name the preview fails to render with error

DagsterInvariantViolationError: new_repo_name not found at module scope in file package/repository.py.

one more branch deployment question: do they shutdown when the PR is merged? not seeing it on mine

Do branch deployments require a K8S cluster or will they run on ECS as well?

hello, I would like to use python to programmatically check failed jobs, the queue, and trigger reruns in dagster-cloud. I should be looking at the GraphQL Python Client? do I also need the dagster-cloud CLI? thanks!

Hello - I have two repos, one on

v1+

and one on

0.15+

with legacy APIs such as

@solid

. Can I upgrade my agent (k8) to

v1+

without messing up my repo on 0.15 and still have any newer functionality available to me?

What's the relationship between a repo/job's specified executor and a Dagster Cloud agent's type? Like if I had a Kubernetes agent running and specified a multiprocessing executor on the job, that would be nonsensical.

If I use EKS on EC2 Spot instances with

k8s_job_executor

, and the spot instances get interrupted, how will the executor respond -- retry the step on a new instance? Treat it as a failure? (Is that something I would expect to be handled at the k8s control plane, or the job executor?)

Hello.... I upgraded my agent to

1.0.3

and wanted to use my existing agent for branch deployments (I have a separate environment than production). However, the env

DAGSTER_CLOUD_DEPLOYMENT_NAME

is showing up as `a725e19f`…

which also happens to be the last random value at the end of my deployment.

(if this helps, I was on the beta and I made no changes except no longer using a dedicated agent for branching)

is there a way to do a dry run locally of the config checks that run when you deploy a code location to cloud?

What's the command for shutting down a k8s agent with helm? 😄

(cross-posting from dagster-support at @sandy's request) Is there any way to determine which dagster-cloud deployment an op is running within? (e.g., dev, prod, or a branch deployment) In particular, I'd like to be able to construct log messages, emails, or other notifications that reference the dagit URL corresponding to the current run id. I see that OpExecutionContext can reference tags (via

context.get_tag

), although the default tags in dagster-cloud don't seem to contain info about the current deployment or dagit URL. I wonder if there might be any relevant info stored as instance-level settings under

context.instance.get_settings

? Or perhaps as environment variables?

i'm not sure TTL is working as expected on branch deployments. these have been up since yesterday afternoon with no activity, but ttlSeconds = 7200

Hey, this has never happened and thought i should flag it. i had two jobs which each trigger a dbt cloud job. i could see a

success

status on dbt cloud, so i know the jobs completed, but dagster jobs were still going on the dagster UI (it keeps polling). We’re running a hybrid deployment on ECS and i saw both ECS tasks had stopped. Therefore, i’m guessing the issue was on dagster-cloud’s end. I’ve clicked

Terminate

and got the following message. I retried with the force option and it took ~15min to finally cancel.

One question around branch deployments: https://docs.dagster.io/guides/dagster/branch_deployments in DBT it is possible to only materialize the changes to the schema of the branch (to not require duplication of the data, and wasting time for complex materializations of a full TBs sized DWH). Dagster seems to prefer to always construct a full copy. And perhaps with Snowflakes cloning this is even feasible. However, in general with other storage engines it will be impossible to change sourcing all the data. I think you somehow will have to be able to support to only write/read changes from the branch`s schema. How should an IO manager potentially handle reading from master but writing to the specific branch name in an intelligent way?

I have a question around (github actions) interacting with dagster cloud. Currently they take way too long for me as the base image is not cached. How can I enable caching? The pipeline looks like this: - on a PR - install all python dependencies (first conda then pip via conda) - the whole conda part is cached! - testing: - validation of code formatting - flake8 linting - mypy type checks - unit tests - dagster cloud deploy preview (dagster-io/dagster-cloud-cicd-action/preview@v0.2.5) - merge/commit to master - auto increment the version number & update release to changelog & push version tag - deploy to dagster cloud using (dagster-io/dagster-cloud-cicd-action/deploy@v0.2.5) The dagster code builds a docker image (from scratch) and therefore it is not cached. - How can enable cachiing (just like it also nicely works with mamba/conda) so the build pipeline gets faster? - IT seems strange that I first install the dependencies and then the dagster util builds another container. How can I somehow work straight inside the container and: - if every test passes publish/forward that same image (already faster than re-building) - enable caching to speed up the builds

The preview is exited (even though I would want to stay in the preview) when clicking on view global asset lineage.

Hi Team! I'm trying to add a code location but I'm running into a failure when Dagster Cloud tries to load it. It looks like the agent is able to successfully create the ECS service and a task is successfully started; however, it looks like the agent is timing out when trying to connect to the code server on the created task. Can anyone suggest how I can troubleshoot this? Deployment details and logs below. Deployment details: • Dagster agent (v1.0.4) deployed on ECS Fargate • Dagster code image deployed to ECR • contents posted in YAML editor when adding code location from Dagster Cloud UI (I've also tried specifying python_file in the code_source section but that resulted in the same error):

location_name: my-test-dagster-location
image: ${MY_AWS_ACCOUNT_ID}.<http://dkr.ecr.us-east-1.amazonaws.com/bu-dagster-development:latest|dkr.ecr.us-east-1.amazonaws.com/bu-dagster-development:latest>
code_source:
  package_name: data_warehouse

• contents of Dockerfile for Dagster code:

FROM python:3.8-slim

ADD requirements.txt .

RUN pip install -r requirements.txt

ADD . .

• contents of requirements.txt for Dagster code (previously, I did not have pinned versions for each; however, I tried this after seeing this suggested in a previous discussion; however, both methods resulted in the same below error):

dagster==0.15.8
dagster-cloud==0.15.8
dagster-airbyte==0.15.8
dagster-aws==0.15.8
dagster-dbt==0.15.8

Will post the stack traces / logs in a reply to this message