Good Morning all! sorry to ask but I need help to get the captured logs of a run with graphql query. I am using a cloud instance and I did not find an example in the docs. Do anyone have an example of this query?

hello, since about 7am PT the majority of sensor ticks in our main deployment have failed due to timeout, these sensors had been healthy up until that point.

hey folks, we're using dagster Cloud Hybrid with Fargate. we sometimes see runs failing to start because of "You’ve reached the limit on the number of vCPUs you can run concurrently". Is there anyway to prevent starting more tasks when we're at our vCPU quota?

Hi. I want to execute processes that branch in pipelines concurrently in parallel. I have the following setup, but if max_concurrent is greater than 2, both processes will stop at the point of branching. Is this an incorrect way to make the branching process? • LaunchPad

execution:
  config:
    max_concurrent: 2
....

• job.py

....
@job(executor_def=multiprocess_executor)
def elt_job():
    .....
    output_1, output_2 = op_a(var=output_1)
    op_b(var=output_1)
    output_3 = op_c(var=output_2)
    ...

• op.py

...

@op
def op_a(context, var: str)
    ....
    return "output_1", "output_2"

@op(ins={"var": In()})
def op_b(context, var: str)
    ....


@op(ins={"var": In()})
def op_c(context, var: str)
    ....
    yield Output(value="output_3", output_name="output_3")

...

Hi everyone. First post so go easy on me. We're using Dagster Cloud in hybrid mode - Dagster Cloud for our UI and scheduling, but with code running via an agent process on a private VM. This is functional but needs manual intervention to redeploy code. We haven't setup CI/CD yet so if there's a serverless option that automatically deploys that would be a shortcut for us. Does anyone have good experience of using any Docker or serverless hosting for agents? We've tried Digital Ocean's App platform but that doesn't support

multiprocessing

required by Dagster.

Hi folks, I’m trying to follow these docs to emit Datadog metric from my Dagster job, but it requires a Datadog agent. Do you have any recommendations for getting a Datadog agent running on the host running our jobs? We run Dagster Cloud with ECS on AWS.

Hi folks, setting up a new deployment on Dagster Cloud and so far it has been working well. However, as I am trying out the backfill functionality I keep running into a partition mapping issue. Pasting the stack trace here from the UI. My data

assets

are structured in a way that I have a dynamic partition definition that is a sparse set of dates in the upstream asset and a monthly time window partition definition in the downstream asset. Has anyone run into issues with partition mappings pertaining to dynamic partitions?

not able to double click a asset on the lineage page

Hi, I’m running a docker container via the

docker_container_op

. I’ve had success doing this locally with

dagster dev

, but it’s been failing on Dagster Cloud. I’ve tried this with both the very simple example of using the busybox container to echo something, and using a private container registry. Both work locally, but fail on Dagster Cloud. The error I get seems to be unrelated to my own code.

dagster._check.CheckError: Failure condition: Couldn't import module dagster_cloud_serverless_agent.serverless.user_code_launcher when attempting to load the configurable class dagster_cloud_serverless_agent.serverless.user_code_launcher.ServerlessUserCodeLauncher

I’ll post the full error in the thread. 🧵 Is Dagster Cloud supposed to be able to support this feature?

hello! how can I use a private library in my cloud, by private I mean a library that isn’t published in PyPi neither is in a public repo. This private library is inside a folder, this folder could be inside the dagster project or outside. thanks! :next-level-daggy:

Hello! I attended the bi-weekly demo last week and our team is specifically interested in a Cloud demo. We're looking into an orchestration tool and are considering Airflow, Prefect, and Dagster. Would someone here be able to setup a demo? I also filled a contact Dagster sales form here

Hello, we encountered a deploy failure late today after quite a few successful deploys earlier in the day. On further investigation we found that

typing-extensions

version differed between the early and late deploys (was

4.5.0

, now

4.6.0

) - dagster calls for

typing_extensions>=4.4.0

. The new version (

4.6.0

) was released between 5-6pm US Pacific Time... changenotes here. Our CI/CD pipeline installs dagster fresh on each run, so there is some reason to believe that one of the changes may not be forward compatible. Pinning

typing-extensions

to

4.5.0

resolved the issue.

Hey, Our dev team are weighing up using dagster-cloud fully serverless vs using a Hybrid deployment. We would like to know more about the data privacy and security when having a fully serverless deployment. Is there any documentation specifically around data privacy, security etc. for serverless specifically? Thanks!

👋 Is there a way to trigger a build & branch deploy via the CLI? I have been able to create a branch deploy with the CLI but am flummoxed on how to actually build and deploy to the branch deploy. I have been successful in building and deploying to

prod

with:

dagster-cloud serverless deploy . --location-name my_location --package-name mypackage

Currently using serverless. Thanks!

Hello! I’m using

submit_job_execution

function for executing jobs in Dagster cloud and I see that the function has a parameter called

run_config

, do you have an example of how can I pass a dictionary or a dataframe or both using that parameter? Thanks!

What is the i.p. address for Dagster cloud. We need to give access to our AWS account -- and want to temporarily use ip-whitelisting.

Hello! I’m having a bad time trying to pass some variables to

submit_job_execution

function.Here is what Im trying to do: https://pastebin.pl/view/50193ecd (pass some parameters to my asset using

submit_job_execution

)whats the correct way to create the dictionary for

run_config

?

👋 Hello, we would like to announce a new version of our GitHub CI/CD workflow for all Hybrid users. Your existing workflows will continue to work and you do not need to port them. However, the new workflow makes a few things better: 1. Full and branch deployments are both implemented in a single file so there is less duplication. 2. The docker build is fully under your control so you can use any advanced features of the build-push-action. 3. Multiple docker builds happen in a single job by default, so the local build cache gets reused. It is also easier to reuse the same docker image tag for multiple code locations. 4. You can easily select a subset of code locations to deploy. The new workflow is also available for Gitlab. Let us know if you want to use the new workflow and have any questions!

Hello! I switched to the Hybrid agent, but the Cloud agent keeps appearing in Branch Deployments (in addition to my k8s agent). The Cloud agent tries to spin up code locations but fails with (as the image is not managed by Dagster Cloud). My k8s agent has the following configuration:

dagsterCloud:
  deployments:
    - prod

  branchDeployments: true

As I understand, it should appear for both

prod

and Branch Deployments, right? Why can this happen?

Hello, I got this error and I want to understand what happened We are on a hybrid setup. Checked the agent logs and I found the same error

when deploying using the command:

dagster-cloud serverless deploy-python-executable ./my-project --location-name whatever --package-name my_project

sometimes I receive this error: https://pastebin.pl/view/5e6d9ebf . Im using poetry environment and my requirements.txt is: https://pastebin.pl/view/3e70ddec

If Dagster Cloud is charging per op now instead of op-minute, would it make more sense for the usage tab to show number of ops executed instead of op-minutes? Or is there a different place to see the number of ops executed in a given month?

Hi everyone, I'm currently using Gitlab CI/CD to deploy my code location. I use the following script : https://github.com/dagster-io/dagster-cloud-action/blob/main/gitlab/serverless-ci.yml As it is a bit slow, I want to try the fast deploys with PEX. Is it possible with Gitlab ? If yes, what do I have to change ? Many thanks

Hi all, hope you're doing well 🙂 I am having a problem with Dagster cloud when I try to choose for serverless deployment in the setup page. I get the error:

You do not have permission to activate serverless deployments.

What should I do to fix this?

Hello! I'm unable to get my github actions to pass due to the below error. Has anyone seen this before?

Error: buildx failed with: ERROR: failed to solve: process "/bin/sh -c if [ -f \"setup.py\" ]; then         pip install .;     fi" did not complete successfully: exit code: 1

Hey all, I switched our dagster cloud instance off of the free trial today, and now I’m getting deployment failures with:

Error: No Dagster Cloud agent is actively heartbeating. Make sure that you have a Dagster Cloud agent running.

Checking the agents window, it looks like we have one inactive agent, and the message ‘Activating serverless deployment’ that’s been spinning for a few hours (screenshot). Can someone help me get this back up and running?

Good afternoon; When trying to run the Cloudformation Stack for the hybrid solution, I get the following error;

Resource handler returned message: "Error occurred during operation 'ECS Deployment Circuit Breaker was triggered'." (RequestToken: 80eb7a80-f711-36a6-3712-2d14c936d93a, HandlerErrorCode: GeneralServiceException)

Is it possible to construct a DagsterInstance which points to a Dagster Cloud deployment from outside of a user code deployment? The use case would be running jobs programmatically from somewhere like AWS Lambda but have the dagster events streamed back to a Dagster Cloud deployment. This would be different from a step launcher in that the trigger to launch the job would be external to Dagster. Sorry if this doesn't make any sense at all, I'm just exploring ideas 💡

Is there a recommended way to set up google application credentials using dagster-cloud? Some of our jobs and assets need to authenticate, but not all of the resources have a credentials argument (e.g. GCSResource). We’ve set up gcp credentials in a dagster environment variable, and our current solution is to decode the base64 credentials key and set the GOOGLE_APPLICATION_CREDENTIALS file and env variable manually, but this seems hacky.