I'm having trouble viewing the end of the run logs in the Dagster Cloud UI. It seems that the scrolling doesn't go all the way to the bottom. I have an error as the last log I can partially see, but I can't see the whole stack trace because I can't scroll any further to where I could usually click the button to show the whole message. It seems that the last log message or two is cut off across all my runs, across all deployments

Dagster Cloud (Hybrid) + Compute Logs question- The docs say that run workers upload all compute logs to Dagster Cloud. This seems like it could be too much data to store - and for privacy reasons, ideally Dagster wouldn’t store all our logs. We currently ingest all stdout/stderr logs into Loki from processes kicked-off by our existing scheduler - so ideally Dagster Cloud’s UI would ask the code deployment to query Loki for logs when it needs to display them (or at least grab them from our own S3 bucket or similar.) How are others dealing with this? Does Dagster Cloud just host its own copy of all your process logs - and you keep another copy of logs in CloudWatch/Loki/Splunk for search, alerting, etc.?

Does anyone experience weird issues with cloud deployments being stuck on installing python dependencies from pip? Did not change any dependencies but seeing the issue since yesterday evening or so.

#13 60.68 INFO: pip is looking at multiple versions of json5 to determine which version is compatible with other requirements. This could take a while.
#13 60.70   Downloading json5-0.9.4-py2.py3-none-any.whl (17 kB)
#13 62.51   Downloading json5-0.9.3-py2.py3-none-any.whl (17 kB)
#13 64.31   Downloading json5-0.9.2-py2.py3-none-any.whl (27 kB)
#13 66.11   Downloading json5-0.9.1-py2.py3-none-any.whl (27 kB)
#13 67.91   Downloading json5-0.9.0-py2.py3-none-any.whl (27 kB)
#13 69.70 INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See <https://pip.pypa.io/warnings/backtracking> for guidance. If you want to abort this run, press Ctrl + C.
#13 69.70 INFO: pip is looking at multiple versions of httptools to determine which version is compatible with other requirements. This could take a while.
#13 69.71 Collecting httptools>=0.5.0
#13 69.73   Downloading httptools-0.6.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (441 kB)
#13 69.74      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 441.7/441.7 kB 63.9 MB/s eta 0:00:00
#13 93.11   Downloading httptools-0.5.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (427 kB)
#13 93.12      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 427.8/427.8 kB 73.1 MB/s eta 0:00:00
#13 116.4 INFO: pip is looking at multiple versions of h11 to determine which version is compatible with other requirements. This could take a while.
#13 116.5 Collecting h11>=0.8
#13 116.5   Downloading h11-0.13.0-py3-none-any.whl (58 kB)
#13 116.5      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.2/58.2 kB 18.3 MB/s eta 0:00:00
#13 186.9   Downloading h11-0.12.0-py3-none-any.whl (54 kB)
#13 186.9      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.9/54.9 kB 18.0 MB/s eta 0:00:00
#13 233.5 INFO: pip is looking at multiple versions of httptools to determine which version is compatible with other requirements. This could take a while.
#13 257.0   Downloading h11-0.11.0-py2.py3-none-any.whl (54 kB)
#13 257.0      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.6/54.6 kB 17.2 MB/s eta 0:00:00
#13 327.9   Downloading h11-0.10.0-py2.py3-none-any.whl (53 kB)
#13 327.9      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.7/53.7 kB 15.9 MB/s eta 0:00:00
#13 351.4 INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See <https://pip.pypa.io/warnings/backtracking> for guidance. If you want to abort this run, press Ctrl + C.
#13 398.8   Downloading h11-0.9.0-py2.py3-none-any.whl (53 kB)
#13 398.8      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.6/53.6 kB 19.0 MB/s eta 0:00:00
#13 469.7   Downloading h11-0.8.1-py2.py3-none-any.whl (55 kB)
#13 469.7      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 55.8/55.8 kB 17.6 MB/s eta 0:00:00
#13 540.8   Downloading h11-0.8.0-py2.py3-none-any.whl (55 kB)
#13 540.9      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 55.6/55.6 kB 17.4 MB/s eta 0:00:00
#13 611.4 INFO: pip is looking at multiple versions of h11 to determine which version is compatible with other requirements. This could take a while.
#13 611.4 INFO: pip is looking at multiple versions of graphene to determine which version is compatible with other requirements. This could take a while.
#13 611.4 Collecting graphene>=3
#13 611.4   Downloading graphene-3.2.2-py2.py3-none-any.whl (125 kB)
#13 611.4      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 125.6/125.6 kB 36.3 MB/s eta 0:00:00
#13 965.5 INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See <https://pip.pypa.io/warnings/backtracking> for guidance. If you want to abort this run, press Ctrl + C.
#13 1178.8   Downloading graphene-3.2.1-py2.py3-none-any.whl (125 kB)
#13 1178.9      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 125.1/125.1 kB 5.8 MB/s eta 0:00:00
#13 1728.2   Downloading graphene-3.2-py2.py3-none-any.whl (124 kB)
#13 1728.3      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 124.6/124.6 kB 6.4 MB/s eta 0:00:00
#13 2279.0   Downloading graphene-3.1.1-py2.py3-none-any.whl (121 kB)
#13 2279.0      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 121.4/121.4 kB 6.0 MB/s eta 0:00:00
#13 2831.6   Downloading graphene-3.1-py2.py3-none-any.whl (114 kB)
#13 2831.7      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 114.2/114.2 kB 6.0 MB/s eta 0:00:00
#13 3389.9   Downloading graphene-3.0-py2.py3-none-any.whl (112 kB)
#13 3389.9      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 112.9/112.9 kB 5.9 MB/s eta 0:00:00
#13 3390.0 INFO: pip is looking at multiple versions of gql[requests] to determine which version is compatible with other requirements. This could take a while.
#13 3390.0 Collecting gql[requests]>=3.0.0
#13 3390.0   Downloading gql-3.4.0-py2.py3-none-any.whl (65 kB)
#13 3390.0      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 65.2/65.2 kB 17.5 MB/s eta 0:00:00
#13 5062.8 INFO: pip is looking at multiple versions of graphene to determine which version is compatible with other requirements. This could take a while.
#13 7319.6   Downloading gql-3.3.0-py2.py3-none-any.whl (63 kB)
#13 7319.6      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 63.4/63.4 kB 3.7 MB/s eta 0:00:00
#13 8449.9 INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See <https://pip.pypa.io/warnings/backtracking> for guidance. If you want to abort this run, press Ctrl + C.

Anyone else just start getting an influx of this error?

dagster_cloud_cli.core.errors.GraphQLStorageError: Error in GraphQL response: [{'message': 'Internal Server Error (Trace ID: 7836303090216615385)', 'locations': [{'line': 4, 'column': 13}], 'path': ['eventLogs', 'getMaterializationCountByPartition']}]

Hello team! 🙋‍♂️ questiongif CustomRunCoordinator & Dagster.Cloud dagsir question questiongif I implemented a custom-run-coordinator which works great locally on

dagster dev

and is configured like this in

dagster.yaml

run_coordinator:
  module: run_coordinator
  class: CustomRunCoordinator
  config:
    tag_concurrency_limits:
      - key: "my_tag"
        value:
          applyLimitPerUniqueValue: true
        limit: 1

I explored almost every bit of the web, trying to figure out how to define my

CustomRunCoordinator

from

project-root/run_coordinator.py

as my dagster.cloud run coordinator, and still haven't figured it out. How can I specify a concrete run coordinator for cloud deployments? (preferably via

dagster_cloud_staging.yaml

/`dagster_cloud_prod.yaml` or other config files and not through the UI/CLI) Thank you very much, folks! keanu thanks 🙏

Hi folks! Trying to make a switch from docker github actions workflow to pex-based, but faced (across everything else) this error with one of our code location:

Still waiting for agent to sync changes to globaldata_pipeline, dataset_export_for_longevity_map, and article_ingestions. This can take a few minutes.
Still waiting for agent to sync changes to globaldata_pipeline, dataset_export_for_longevity_map, and article_ingestions. This can take a few minutes.
Still waiting for agent to sync changes to globaldata_pipeline, dataset_export_for_longevity_map, and article_ingestions. This can take a few minutes.
Still waiting for agent to sync changes to globaldata_pipeline, dataset_export_for_longevity_map, and article_ingestions. This can take a few minutes.
Still waiting for agent to sync changes to globaldata_pipeline, dataset_export_for_longevity_map, and article_ingestions. This can take a few minutes.
Still waiting for agent to sync changes to globaldata_pipeline, dataset_export_for_longevity_map, and article_ingestions. This can take a few minutes.
Still waiting for agent to sync changes to globaldata_pipeline, dataset_export_for_longevity_map, and article_ingestions. This can take a few minutes.
Still waiting for agent to sync changes to globaldata_pipeline, dataset_export_for_longevity_map, and article_ingestions. This can take a few minutes.
Still waiting for agent to sync changes to globaldata_pipeline, dataset_export_for_longevity_map, and article_ingestions. This can take a few minutes.
Still waiting for agent to sync changes to globaldata_pipeline, dataset_export_for_longevity_map, and article_ingestions. This can take a few minutes.
Still waiting for agent to sync changes to globaldata_pipeline, dataset_export_for_longevity_map, and article_ingestions. This can take a few minutes.
Error: Some locations failed to load after being synced by the agent:
Error loading globaldata_pipeline: {'__typename': 'PythonError', 'message': 'Exception: Timed out after waiting 315s for server globaldatapipeline-fe2bfeea94c5a62adf63e1027a37f39372-b8a41f.serverless-agents-namespace-1:4000.\n\nTask logs:\n    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable\n    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\ngrpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = "failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/tmpn1b8it8p: No such file or directory"\n\tdebug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/tmpn1b8it8p: No such file or directory {grpc_status:14, created_time:"2023-11-20T07:50:49.169683184+00:00"}"\n>\n{"time": "20/Nov/2023:07:50:51 +0000", "log": "registry - Ignoring failure to clean up local unused path /tmp/pex-files/source-948ebbb361e2d784a6ec5b1c21065161f43621a3.pex\\nTraceback (most recent call last):\\n  File \\"/dagster-cloud/dagster_cloud/pex/grpc/server/registry.py\\", line 193, in cleanup_unused_files\\n    os.remove(path)\\nFileNotFoundError: [Errno 2] No such file or directory: \'/tmp/pex-files/source-948ebbb361e2d784a6ec5b1c21065161f43621a3.pex\'", "status": "ERROR", "logger": "root"}\n{"time": "20/Nov/2023:07:50:54 +0000", "log": "_server - Exception calling application: <_InactiveRpcError of RPC that terminated with:\\n\\tstatus = StatusCode.UNAVAILABLE\\n\\tdetails = \\"failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/tmpn1b8it8p: No such file or directory\\"\\n\\tdebug_error_string = \\"UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/tmpn1b8it8p: No such file or directory {grpc_status:14, created_time:\\"2023-11-20T07:50:49.169683184+00:00\\"}\\"\\n>\\nTraceback (most recent call last):\\n  File \\"/usr/local/lib/python3.11/site-packages/grpc/_server.py\\", line 552, in _call_behavior\\n    response_or_iterator = behavior(argument, context)\\n                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\"/dagster-cloud/dagster_cloud/pex/grpc/server/server.py\\", line 172, in Ping\\n    return self._query(\\"Ping\\", request, context)\\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\"/dagster-cloud/dagster_cloud/pex/grpc/server/server.py\\", line 137, in _query\\n    return client_or_error._get_response(api_name, request)  # noqa: SLF001\\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\"/dagster/dagster/_grpc/client.py\\", line 140, in _get_response\\n    return getattr(stub, method)(request, metadata=self._metadata, timeout=timeout)\\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\"/usr/local/lib/python3.11/site-packages/grpc/_channel.py\\", line 1161, in __call__\\n    return _end_unary_response_blocking(state, call, False, None)\\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\"/usr/local/lib/python3.11/site-packages/grpc/_channel.py\\", line 1004, in _end_unary_response_blocking\\n    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable\\n    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\ngrpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:\\n\\tstatus = StatusCode.UNAVAILABLE\\n\\tdetails = \\"failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/tmpn1b8it8p: No such file or directory\\"\\n\\tdebug_error_string = \\"UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/tmpn1b8it8p: No such file or directory {grpc_status:14, created_time:\\"2023-11-20T07:50:49.169683184+00:00\\"}\\"\\n>", "status": "ERROR", "logger": "grpc._server"}\n{"time": "20/Nov/2023:07:50:59 +0000", "log": "registry - Ignoring failure to clean up local unused path /tmp/pex-files/deps-f3b8ee6b2caf2a7277819e380fa17e3221a7fd25.pex\\nTraceback (most recent call last):\\n  File \\"/dagster-cloud/dagster_cloud/pex/grpc/server/registry.py\\", line 193, in cleanup_unused_files\\n    os.remove(path)\\nFileNotFoundError: [Errno 2] No such file or directory: \'/tmp/pex-files/deps-f3b8ee6b2caf2a7277819e380fa17e3221a7fd25.pex\'", "status": "ERROR", "logger": "root"}\n\nMost recent connection error: dagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server. gRPC Error code: UNAVAILABLE\n\nStack Trace:\n  File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1710, in _wait_for_server_process\n    client.ping("")\n  File "/dagster/dagster/_grpc/client.py", line 200, in ping\n    res = self._query("Ping", api_pb2.PingRequest, echo=echo)\n  File "/dagster/dagster/_grpc/client.py", line 167, in _query\n    self._raise_grpc_exception(\n  File "/dagster/dagster/_grpc/client.py", line 150, in _raise_grpc_exception\n    raise DagsterUserCodeUnreachableError(\n\nThe above exception was caused by the following exception:\ngrpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = "failed to connect to all addresses; last error: UNKNOWN: ipv4:10.0.115.152:4000: connection attempt timed out before receiving SETTINGS frame"\n\tdebug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:10.0.115.152:4000: connection attempt timed out before receiving SETTINGS frame {grpc_status:14, created_time:"2023-11-20T07:53:53.102027102+00:00"}"\n>\n\nStack Trace:\n  File "/dagster/dagster/_grpc/client.py", line 165, in _query\n    return self._get_response(method, request=request_type(**kwargs), timeout=timeout)\n  File "/dagster/dagster/_grpc/client.py", line 140, in _get_response\n    return getattr(stub, method)(request, metadata=self._metadata, timeout=timeout)\n  File "/usr/local/lib/python3.10/site-packages/grpc/_channel.py", line 1161, in __call__\n    return _end_unary_response_blocking(state, call, False, None)\n  File "/usr/local/lib/python3.10/site-packages/grpc/_channel.py", line 1004, in _end_unary_response_blocking\n    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable\n\n', 'stack': ['  File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1304, in _reconcile\n    self._wait_for_new_server_ready(\n', '  File "/dagster-cloud/dagster_cloud/workspace/ecs/launcher.py", line 461, in _wait_for_new_server_ready\n    self._wait_for_dagster_server_process(\n', '  File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1688, in _wait_for_dagster_server_process\n    self._wait_for_server_process(\n', '  File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1723, in _wait_for_server_process\n    raise Exception(\n']}

Any ideas what can it be? Does seems to be connected with our code but I'm not quite sure however.

Sorry for spamming here, but got more and more question the further I dig into actions pipeline. However this one is pretty broad one. So basically the issue is pex-based pipeline run time. Before that we have old docker-based pipeline with 3 code locations. All locations have same dependencies and were built in about 10 minutes each. So in total the whole deployment took like 11 minutes or so since locations were build in parallel in different jobs. However after switch to pex fast deploy, the pipeline takes like ~30 min to run. It looks like it always rebuild all code locations independent on what code changes were made, were dependencies changed and so on. And it also basically builds the exact same set of dependencies for all three code location taking x3 time for it. Is there any kind of magic I missed to make it works actually fast and not rebuild everything from scratch every time? Just a wild guess, that I have to merge it into master and take a look how further deployments will perform on top of it, but I'm not sure will it actually work and don't want to merge huge (possible worsening) changes without confirmation :)

Hey I'm sorry if this question seems trivial but I'm confused about how I should clean up my Dagster Cloud Serverless repo. Right now I am starting basically from scratch, having only deployed the Hackernews example on my Github: From here, can anybody provide advice or resources on: 1. Cleaning up Dagster Cloud to not have the example in the UI? 2. How should I go about building my Dagster pipeline? (Should I clone the existing repo to Local, make changes and push)? Maybe I missed something

Hey everyone, having an issue with deploy-python-executable. It gets stuck at this point, and then takes hours to complete:

dagster-cloud serverless deploy-python-executable . \
  --location-name ciro \
  --package-name etl_pipelines \
        --python-version=3.11
Going to deploy location ciro
Building Python executable for ciro from directory /Users/XXXX/etl and Python 3.11.
Building project dependencies for Python 3.11, writing to /var/folders/pj/zxnmgbk523ldprrb4bm9rgbw0000gn/T/tmp5yu67pe7
Warning: Failed to build dependencies in current environment:ValueError: No pre-built wheel was available for pendulum 2.1.2.No pre-built wheel was available for pendulum 2.1.2.
Warning: Falling back to build in a docker environment
Running docker <http://ghcr.io/dagster-io/dagster-manylinux-builder:latest|ghcr.io/dagster-io/dagster-manylinux-builder:latest>
Mapped folders:
 - /var/folders/pj/zxnmgbk523ldprrb4bm9rgbw0000gn/T/tmp5yu67pe7 -> /output

Please let me know what other information might be helpful to figure out what’s going on, and any help regarding this problem would be greatly appreciated.

I'm considering a hybrid deployment in GCP; is the docker agent the right one to use? we use Cloud Run extensively, fwiw - so if there were something that integrated more closely with that, it would be interesting

I'm having some trouble getting my DBT repo to show up inside the Docker container with a Docker deployment (i.e., fast deployment = false) on Serverless. I'm following these instructions to get

dbt deps

to work, however it seems the dbt repo is not being copied into the Docker container so this step fails. Any help would be appreciated! (My workflow files and logs are shown in the thread.)

The Dagster Agent in my severless deployment is constantly going down (I see a

Activating serverless deployment

in the

Deployment > Agents

tab every few minutes) and my runs are therefore unable to execute, constantly giving a

Still waiting for compute resources to spin up.

message and then eventually giving the following error and failing:

Run dequeue failed to reach the user code server after 1 attempts, failing run
dagster._core.errors.DagsterUserCodeUnreachableError: Timed out waiting for call to user code LAUNCH_RUN

I don't see any mention of outages on the Dagster Cloud Status page. Why is the agent so...unstable(?) and constantly redeploying? Is this a common occurrence for serverless users?

I'm having trouble deploying dagster project using the

dagster-cloud

cli. The deployment hasn't changed. I suspect it's related to importing a local package as a library in the setup.py:

install_requires=[
        # Local dependencies
        "dagster-lib @ file://"
        + os.path.join(os.path.dirname(__file__), "../libs/dagster_lib"),

I get:

dagster._core.errors.DagsterImportError: Encountered ImportError: `cannot import name 'SqlIOManager' from 'dagster_lib.resources.io_managers' (/venvs/1a7e8cb1c6a5/lib/python3.10/site-packages/dagster_lib/resources/io_managers.py)` while importing module data_pipelines. Local modules were resolved using the working directory `/venvs/4c52386641a1/lib/python3.10/site-packages/working_directory/root`. If another working directory should be used, please explicitly specify the appropriate path using the `-d` or `--working-directory` for CLI based targets or the `working_directory` configuration option for workspace targets.

In the deployment logs I notice:

Reusing cached dependencies: deps-dabf9cc84304f18af0dce6c9b35b3c8730259878.pex

Is it possible this dependency is cached and not update? Is there anything else that might cause this?

I'm having a problem deploying to Dagster Cloud using the CLI. Each time I run this:

dagster-cloud serverless deploy-python-executable ./dagster_university \
  --location-name dagster_university \
  --package-name dagster_university

I get a

ModuleNotFoundError

error like this:

Going to deploy location dagster_university
Building Python executable for dagster_university from directory /Users/john/Projects/dagster_essentials/dagster_university and Python 3.8.
<<I snipped the traceback>>
ValueError: Error running setup.py egg_info: Traceback (most recent call last):
  File "/Users/john/Projects/dagster_essentials/dagster_university/setup.py", line 1, in <module>
    from setuptools import find_packages, setup
ModuleNotFoundError: No module named 'setuptools'

Hey all! I'm trying to move to a Hybrid Cloud deployment and I'm trying to understand good practices around deploying code/images. My understanding of running an agent with my custom code is to: • Use CI/CD to create an image with the latest code I want to run (i.e. the asset definitions etc) • Specify this image in

dagster_cloud.yaml

However, the example reference specifies a docker image using the

latest

tag, which is not best practice in Docker as far as I know. If I wanted to specify e.g. a git commit hash as the docker image tag, is there a mechanism for that? Basically I want the code the agent uses to update whenever I make a new commit. I can't set the

dagster_cloud.yaml

to specify the current commit as the tag because I don't know the commit hash before committing, so it's a circular dependency. I've looked for about an hour in various docs and couldn't find any good examples. Could someone point me in the right direction?

Hello, We are using Dagster Cloud to manage our infrastructure. Since this morning, the performance has been very poor. Even our health check job, which does nothing but

return 1

, takes more than 5 minutes to initiate the Run, and half the time, it ends in a timeout with the error

dagster._core.errors.DagsterUserCodeUnreachableError: Timed out waiting for call to user code LAUNCH_RUN

after several minutes of running (even though no other job or sensor is running in parallel). Launching other partitioned jobs results in 504 errors before even a single Run is initiated. How can we debug this kind of issue?

I realize I probably should have put the following message in this channel instead of support: Hey all, is it normal for a run to need at least 1 minute to allocate resources before it can start running? Anyway to make that faster? Would love to use Dagster but not sure if it can be used for real time data orchestration. I presume if we host Dagster on our own server, it's not an issue, but I'd prefer to use Dagster Cloud.

Howdy! I have a graphql query using python requests library to get the list of current deployments. Everything is working fine when I run this locally, when I attempt to run through a github action it fails with 401 - Unauthorized. Is there anything that would prevent a user cloud api token from being able to query using a Github Action Runner? My github action setup is using the same token, url, query, variables, etc as the local execution

Hello, I am having issues deploying to Dagster Cloud (using dagster cloud action 0.1.31 with gitlab). The exact same code seemed to be working yesterday but now I have this error. Any idea of what it could be ? Thank you

As of a bit over 24 hours ago, my automaterialize evaluations have gone from taking a few seconds each to running on for 1.5 hours - screenshot attached. I haven't pushed any new code for about 4 days. The code location reloaded at about the time this began, I'm suspecting it's something to do with the 1.5.10 release which was notified around this time. I've got an impaired production environment as a result - can someone investigate urgently please?

Im seeing a lot of errors in the console when navigating to the overview page of our Dagster Cloud. No partitions details are loading and Im not able to select the materialization configuration at all. Lots of Apollo/ GraphQL errors which would explain data missing from the UI. Is this related to the 1.5.10 release? This is all from

<instance>.dagster.cloud/prod/overview/jobs

when attempting to launch a backfill run with a specific partition configuration

Hey all, I am using Dagster Cloud and would like to use python 3.11.5 instead of the default python 3.8. In your doc this command is provided:

dagster-cloud serverless deploy --location-name=my_location --python-version=3.9

However, when trying to run it, it seems that the

--python-version

argument is not available anymore, could you help me?

I am currently using Dagster Cloud to get files from s3 and my security team is not a big fan of using aws

secret and access

keys and would like to know whether it would be possible for Dagster to assume a specific role in our account? Or if it is planned to be implemented in the near future?

Hello everyone! Leah Padgett, Product Designer at Dagster Labs here. The team is working on a new feature we're currently calling an observability dashboard. The goal is for Dagster Cloud users to be able to monitor, observe and take action all from one convenient place in Dagster. We'd love to hear your thoughts and input on what kind of metrics you'd like to see on the dashboard, as well as how you might use it. Here is a link to a quick survey that should take less than 10 minutes to complete. Also attached is a mockup of a very early design concept for this for your reference. Thanks so much for your help!

Could I get some help with a user management task? I'm on the solo plan but need to change my email. I was going to briefly move up to the team plan, add the new email, switch admin to it, delete the current email, then revert back to solo. Would this work or would I be stuck then in the team plan?

I setup my assets definition like this, On my local machine, all 3 type of assets show up (airbyte, dbt and dagster). But when I deploy the code to dagster-cloud, only the native dagster asset shows up. I followed this template to deploy to dagster cloud: https://github.com/dagster-io/dagster-cloud-hybrid-quickstart

Hi, Today, we're encountering a very annoying problem on dagster-cloud. Every time a deployment is launched via the github action, we have a 50% chance of the deployment going wrong (whatever the code change). What's really annoying is that when the deployment doesn't work, dagster is completely down and no more jobs run until the deployment finally works by running retry after retry. Is there something that can be done about it?

Hello! I would like to have Dagster health information ported to Datadog, so I can get alerts when any of the daemons is unhealthy and also, why not, display that health information in our team dashboard in Datadog. Has anyone succeeded in doing this? And also is such thing somehow easily supported within Dagster or Dagster cloud? From my understanding the library

dagster-datadog

just works within ops and the existence of this Github issue doesn't keep my hopes very high on this matter.

Hey all, I'm trying to connect to my snowflake instance using DBT and keypair authentication from dagster cloud. I get the error that

'Could not deserialize key data. The data may be in an incorrect format, it may be encrypted with an unsupported algorithm, or it may be an unsupported key type (e.g. EC curves with explicit parameters).', [<OpenSSLError(code=109052027, lib=13, reason=123, reason_text=header too long)>]

The keypair does work when I use it to connect using the SnowSQL CLI instance (when referencing the file rather than using the private key directly). Currently I have the private key set as an environment variable in dagster cloud.. which I think is the reason I'm facing the error above. I should instead refer to the key location. If that's the case my question is, is it possible to have the encrypted private key stored in a location on the dagter cloud instance so that I can reference the path?

Hello, how I do I generate Airbyte assets using dagster cloud and CLI. dagster_airbyte.load_assets_from_airbyte_instance I have ran the python script but nothing happened... I have imported dbt project from github but do not find a way to add the airbyte connections. Thank you for your help. I am complete newbie with Dagster