Hi 👋 Is it possible to request GPU instances for specific assets/ops/jobs (assuming a hybrid deployment)? I'm interested in using Dagster to orchestrate training jobs for deep learning models that require hardware acceleration on AWS. I'm aware of the

ecs/cpu

and

ecs/memory

tags for an ECS deployment with Fargate, but it doesn't seem to be possible to request specific EC2 instances within the Dagster context. Is using Kubernetes and per-job/op tags the recommended solution currently?

Hi Dagster team, been having some issues loading usage on the cloud, seems to time out and I get an empty screen.

Hey 👋 We're getting a

sqlalchemy.exc.OperationalError: (psycopg2.errors.QueryCanceled) canceling statement due to statement timeout

error from one of our sensors that's iterating over a GCS bucket and generating a run request for each file. We have a serverless cloud deployment. Is there a way to increase the statement timeout in Dagster Cloud? Traceback:

File "/dagster/dagster/_daemon/sensor.py", line 489, in _process_tick_generator
    yield from _evaluate_sensor(
  File "/dagster/dagster/_daemon/sensor.py", line 626, in _evaluate_sensor
    existing_runs_by_key = _fetch_existing_runs(
  File "/dagster/dagster/_daemon/sensor.py", line 740, in _fetch_existing_runs
    runs_with_run_keys = instance.get_runs(filters=RunsFilter(tags={RUN_KEY_TAG: run_keys}))
  File "/dagster/dagster/_utils/__init__.py", line 697, in inner
    return func(*args, **kwargs)
  File "/dagster/dagster/_core/instance/__init__.py", line 1500, in get_runs
    return self._run_storage.get_runs(filters, cursor, limit, bucket_by)
  File "/dagster-cloud-backend/dagster_cloud_backend/storage/host_cloud/run_storage/storage.py", line 449, in get_runs
    rows = self._readall(query)
  File "/dagster-cloud-backend/dagster_cloud_backend/storage/host_cloud/cloud_storage/mixin.py", line 31, in _readall
    return conn.execute(query).fetchall()
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1380, in execute
    return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement
    ret = self._execute_context(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
    self._handle_dbapi_exception(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2124, in _handle_dbapi_exception
    util.raise_(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/local/lib/python3.8/site-packages/ddtrace/contrib/dbapi/__init__.py", line 138, in execute
    return self._trace_method(
  File "/usr/local/lib/python3.8/site-packages/ddtrace/contrib/psycopg/patch.py", line 70, in _trace_method
    return super(Psycopg2TracedCursor, self)._trace_method(
  File "/usr/local/lib/python3.8/site-packages/ddtrace/contrib/dbapi/__init__.py", line 107, in _trace_method
    return method(*args, **kwargs)

Hi, we currently use Dagster Cloud and I wanted to know if Dagster supports triggering a job in ECS via the AWS integration (https://docs.dagster.io/_apidocs/libraries/dagster-aws#ecs). The documentation doesn’t really clarify or give much idea on if this is possible, and an example of how? If it does, does it also get log information and show it within Dagster?

Hi, I'm having some issues successfully adding billing info -- could someone maybe jump on a DM with me to try and resolve?

Hey all, has something changed in dagster serverless deploys? We’re suddenly getting 401 errors. It seems like the issue may be that the graphql api is resolving to: https://dagster.cloud/prod/graphql instead of <https://l.dagster.cloud/prod/graphql%7Chttps://&lt;our_company&gt;.dagster.cloud/prod/graphql> Nothing has changed in our github actions files, and it was working last friday without any issue. full error stack in 🧵

Hi guys, Is there any way to share resources and/or assets in more than 1 code location? For example I have this in dagster_cloud.yml

locations:
  - location_name: data_analytics
    code_source:
      package_name: analytics
  - location_name: ml
    code_source:
      package_name: machine_learning

and my folder structure

dagster_project
|--- shared_resources
|   |--- resources
|--- analytics
|   |--- assets
|   |--- __init__.py
|--- machine_learning
|   |--- assets
|   |--- __init__.py

and for the defs in dagster_project/analytics/__init__.py and dagster_project/machine_learning/__init__.py I am importing the resources from shared_resources folder

from ..shared_resources.resources import RESOURCES_LOCAL, RESOURCES_PROD, RESOURCES_STAGING
from .assets import source_system_assets

deployment_name = os.environ.get("DAGSTER_DEPLOYMENT", "local")
resources_by_deployment_name = {
    "local" : RESOURCES_LOCAL,
    "staging": RESOURCES_STAGING,
    "prod" : RESOURCES_PROD
}

all_assets = [*source_system_assets]

defs = Definitions(
    assets = all_assets,
    resources = resources_by_deployment_name[deployment_name],
)

But I am having this error. Is it the right way to organise code locations? how do I add more than 1 code location and share some common elements like resources or assets? Thank you!!!

Hi team, I’m interesting in a standard serverless package. I tried calculating the price but I’m not sure if I did it right, here is my calculation: 30 jobs: each job takes 1 minute to finish So, It will cost me (per month) -> 30*1*30*0.04 = $36. Am I right? Or are there any charges that I missed?

Hi all, is there an up-to-date documented list of IP addresses for Dagster Cloud's serverless offering? The closest I've found so far is this: https://discuss.dagster.io/t/8607467/hey-dagster-team-could-someone-please-send-the-list-of-dagst

Morning. Are there non profit discounts available for Dagster cloud?

Hi,

ModuleNotFoundError: No module named 'dagster_cloud_serverless_agent'

getting this error on job run in dagster cloud the job is working in local dev I have the package in setup.py

install_requires=[
        "dagster==1.1.20",
        "dagster-cloud==1.1.20",
        "dagster-docker==0.17.20",
        "dagster-slack==0.17.20",
        "boto3==1.26.81",
    ],

Hello, in dagster cloud hybrid, the first time a worker job is started, it takes almost 3 minutes. Is there anyway to cut it short?

Hi, i am in my very first steps with dagster and dagster-cloud. I am still trying to just get something simple to run on snowflake. Right now i am trying this snippet https://docs.dagster.io/_apidocs/libraries/dagster-snowflake#resource. The deployment to dagster-cloud works but since there is not asset in the definition the assets are emtpy. The job doesn't show up. Is it possible to run something in dagster-cloud without an asset? Just ops?

Hi, we recently implemented a schedule that should create roughly 250 runs. We're getting the following error from our dagster cloud agent container. My understanding is that the error is coming from the cloud agent container requesting the schedule information from the code server container. Why would it be timing out? Our user code server container is not surfacing any logs. It works locally when creating the same runs through the scheduler. How can we debug further where the slowdown is?

dagster._core.errors.DagsterUserCodeUnreachableError: User code server request timed out due to taking longer than 60 seconds to complete.

Is it possible to manage Dagster alerts through Pulumi (or Terraform)? I see in the Dagster alert docs that there's mention of setting them via API, but I haven't come up with IaC references.

For Dagster Cloud, I see that all the logs are coming up as errors in k8s logs.. this for all the pods

Hi, our code deployment suddenly failed and started giving this DagsterUserCodeUnreachableError error. It was working fine until an hour ago and there have been no updates to the docker image. The agent pod is giving this error in the logs.

👋 Hello all! Are there any differences between local execution and executing via the serverless agent? I’ve written a step launcher that works flawlessly when launched on my local machine, but hangs when ran out of the cloud agent. Specifically, while waiting for a SSH channel. EDIT: Was able to reproduce issue in other environment, doesn’t seem dagster related after all 🙂

Hi Dagster team, Context : I'm on Dagster Cloud Serverless. When i run the job from the launchpad, i get the following errors (see screenshot)

The limitations should be 16Go of memory : should be more of enough for the job i want to execute).

As suggested on the error message, there might be a run memory configuration ? But I don't seem to see it anywhere when looking the docs.

PS : I realize that there was a TRY / EXCEPT block statement on my job code . But it did not execute on that error , Do you have any idea of ways to handle exceptions on Dagster Jobs ? (if the job gets killed by an issue outside the python scope , as in the curent example with memory issue)

hi, when i want to add a new code location, I encountered this error, what should I do to make it recover to normal?

hello - i'd like to expose an environment variable for my

setup.py

to use, but it seems that when I pass

--env

to the dagster-cloud CLI it is not set until after setup runs... is that correct? What are the alternatives?

hi y'all! we're looking at setting up Dagster Cloud Hybrid via the "new VPC" CloudFormation stack, and just wanted to clarify some of the required permissions. 🧵

Hi team, do we have the email of the current logged in user as a reserved variable in dagster cloud? My usecase case is to be able to send a message to the user who triggers a run based on a certain criteria

Has anyone used Dagster to coordinate Snow Pipes? Or to launch Terraform jobs? We'd like to have snow pipes with some version control (cheaper than fivetran). The main options we've been recommended is using terraform + snowpipes. I'm not sure if Dagster has support for this type of set-up

👋 hey all, can someone send an up-to-date documented list of IP addresses for Dagster Cloud’s serverless offerings? In a shared DM with myself and @Chris Terrile — this is per the recommendation from the previous message above, happy to file a ticket or go through another flow if preferred: https://dagster.slack.com/archives/C02LJ7G0LAZ/p1677762326303069

Hello, how do I proceed from here? I'm trying to quickstart_aws example with Dagster Cloud but it says it can't find my key...

When using the docker agent for dagster-cloud the GRPC server cannot be found:

Most recent connection error: dagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server. gRPC Error code: UNAVAILABLE

Hey 👋 Does the enterprise instance of Dagster Cloud have an SLA for availability and customer support?

Hi! Is it usual for the deploy github action for a serverless cloud to take over an hour? I'm trying to deploy for the first time.

hi 👋 any idea why our Dagster Hybrid deployment would fail with this time out error? Our ECS logs show that the target service has

Started Dagster code server for file dagster/repository.py on port 4000 in process 1