is there a way to utilize fast deploys when I need to run

dbt deps

to install my dbt packages? We're currently on the docker deployment with a post install script

A few questions about the Serverless spin up times: 1. Is there an SLA or otherwise a known maximum time tasks will take to spin up? Based on my experience so far, the lag is around about 2 mins on average. 2. Are there any plans to reduce the task spin up time at all? It seems rather slow and it would be great if this was at least under a minute

Hi all! I have been running multiple instances of dagster for a few years now. I've created docker compose files which are used to build docker images and then spin up docker containers. I have a container per dagster "component" (daemon, webserver, postgres, repo), all running on the same VM (which is always up). It's time to start thinking about being able to scale up the workload ability, i.e. I'd like to be able to distributed runs of multiple concurrent jobs. It would be ideal to be able to serverlessly spin up a new container for each job. Not all jobs are equal though - some jobs might require minimal compute, but some might need GPU. The same docker image could be used for every job. I have seen https://docs.dagster.io/deployment/guides/aws#launching-runs-in-ecs, which is along the lines of what I'm thinking. We use GCP where we can, but will need to remain cloud agnostic, or at least be able to support the big three. If any of this resonates with anyone or if you have any pointers, please let me know.

Hey dagster team im getting this weird error on dagster serverless I haven't seen it before:

botocore.exceptions.ClientError: An error occurred (ThrottlingException) when calling the ListTasks operation: Rate exceeded
  File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1256, in _reconcile
    self._wait_for_new_multipex_server(
  File "/dagster-cloud/dagster_cloud/workspace/ecs/launcher.py", line 414, in _wait_for_new_multipex_server
    task_arn = self.client.wait_for_new_service(
  File "/dagster-cloud/dagster_cloud/workspace/ecs/client.py", line 497, in wait_for_new_service
    return self.check_service_has_running_task(
  File "/dagster-cloud/dagster_cloud/workspace/ecs/client.py", line 557, in check_service_has_running_task
    running = self.ecs.list_tasks(
  File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 535, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 980, in _make_api_call
    raise error_class(parsed_response, operation_name

hey all since the docker ecs plugin is being deprecated in November of this year, is there an official Dagster statement or guide on how to transition away from the soon-to-be-deprecated tool?

Hi dagster team! I’m having this error while trying to deploy changes to our assets:

Copy
botocore.exceptions.ClientError: An error occurred (ThrottlingException) when calling the DescribeTasks operation: Rate exceeded

  File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1256, in _reconcile
    self._wait_for_new_multipex_server(
  File "/dagster-cloud/dagster_cloud/workspace/ecs/launcher.py", line 414, in _wait_for_new_multipex_server
    task_arn = self.client.wait_for_new_service(
  File "/dagster-cloud/dagster_cloud/workspace/ecs/client.py", line 497, in wait_for_new_service
    return self.check_service_has_running_task(
  File "/dagster-cloud/dagster_cloud/workspace/ecs/client.py", line 587, in check_service_has_running_task
    task = self.ecs.describe_tasks(
  File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 535, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 980, in _make_api_call
    raise error_class(parsed_response, operation_name)

Bug(?) with

AWS_REGION

env var being overridden in Serverless

Hey hello everyone. Can someone tell me if we pay for compute while we wait for the

retry_policy

of an asset? I need to query API that are not very reliable (The data can show up with 1 hour to 3 days of delay), and was wondering if I configure my

retry_policy

to something like 1 or 2 days, would I pay for those days while doing nothing since my

run

is technically in progress?

Hey dagster team could i get more information on this run? https://artemis.dagster.cloud/prod/runs/f54d5a99-9bfe-4c7c-801b-fe2ede45ced1 For whatever reason this job hanged for over 24+ hours. The next job the day after (since its a daily job) finished within 2 hours.

any update on: https://dagstercloud.statuspage.io/? We've had all failing jobs for the last 2 and a half hours.

Do we know what happened yesterday? The impact was non-negligible for us.

Quick question about pricing. Are we getting billed per materialization? I’m trying to change my code to store smaller pieces in

partitions

to keep track of them. T hat would create something around 500k partitions for that specific asset growing at a rate of 1000 partitions per day. Would that highly increase the cost of Dagster Serverless even if the compute time if very small?

The auto-deploy from main is cancelling every time I merge code down. I have to create a new empty commit to force the push to production. Can you help explain what may be going on here?

Hi Dagster, was at Big Data London and I asked Tim about a strange failure we experienced a few days on some jobs running on Serverless. I was told it was due to an AWS outage. As I understand, the retry policy we apply only applies to if there’s an error in the actual code that is running. The the scenario of where a job fails were Dagster could not spin up any infrastructure so our retry policy would not kick in.

Allocating compute resources to ensure run isolation. This can take a few minutes.
Still waiting for compute resources to spin up.
Still waiting for compute resources to spin up.
...
Still waiting for compute resources to spin up.
Agent b48510b6 detected that run worker failed.

In terms of a better experience around this, I believe that Dagster should catch this specific exception and have some type of retry mechanism in place. It feels lazy that Dagster tried once, encountered this failure and then gave up so quickly. If our impacted job was a very important daily job, it could have been much more disruptive.

Hey folks, any other users of serverless would sometimes see jobs not execute for no reason:

How do i properly call envs for my dbt project? They are defined in UI with scope for all code locations, but im still getting

23:35:51  Encountered an error:
Parsing Error
  Env var required but not provided: 'SNOWFLAKE_ACCOUNT'
Error: Process completed with exit code 2.

Got my envs like account: "{{ env_var('SNOWFLAKE_ACCOUNT') }}" in profiles.yml. Serverless job fails on Parse dbt project step

Noticed something odd about a job on Dagster serverless. It failed to carry out the next op in the DAG even though the upstream job was successful. The job runs hourly, and subsequent runs were fine as can be seen in the screenshots.

Step custom_dbt_run_op finished without success or failure event. Downstream steps will not execute.

However you can clearly see in the log output that it was successful blob confounded Is this a bug? Has it been fixed/resolved in a newer version? Current Dagster version

1.4.6

Is it possible to extract job runs statistics out of Dagster to analyse them if we are using Dagster Serverless? Same question if we are not 🙂

Is someone aware of a breaking change in the last version of Dagster or dagster-snowflake? Everything was working perfectly until 1 hour ago. And now the Dagster project won’t build with error

oscrypto.errors.LibraryNotFoundError: Error detecting the version of libcrypto

Hi dagster team, I'm currently facing similar issue to this one : https://dagster.slack.com/archives/C03SMUY0SFQ/p1697125060031739 > oscrypto.errors.LibraryNotFoundError: Error detecting the version of libcryptoµ Therefore, it seems like the proposed fix require to put > ENABLE_FAST_DEPLOYS: 'false' Which is not possible in my case since my deployment depend on a workaround that require ENABLE_FAST_DEPLOYS to true. It's currently affecting our production, could you help out with this ?

Hey I’m trying to

import a dbt project

using the feature in

Deployment -> Add code location -> Import a dbt project

. And I’m not sure what to do. My security team tells me that all grants should be available for Dagster to see m DBT repo, but Dagster Cloud Serverless is not able to find it. Is there something that need to be in that repo in order for Dagster Cloud Serverless to see it? Or a basic DBT Repository is fine? Or do we need a DBT repo wrap with Dagster in order to be able to import it? Thank you very much. Have a great day.

This is a clone of https://dagster.slack.com/archives/C04CW71AGBW/p1697635819979539 Based on @rex comment this was not in the right channel. Sorry for reposting. I’ve received and unexpected

Read Timed out

error while building DBT models with Dagster Cloud Serverless on Snowflake. Is that a known issue or should I create a Dagster issue for it? Here’s the complete stack trace.

dagster_cloud_cli.core.errors.GraphQLStorageError: HTTPSConnectionPool(host='figment.agent.dagster.cloud', port=443): Read timed out. (read timeout=60)
  File "/usr/local/lib/python3.8/site-packages/dagster/_core/execution/plan/execute_plan.py", line 273, in dagster_event_sequence_for_step
    for step_event in check.generator(step_events):
  File "/usr/local/lib/python3.8/site-packages/dagster/_core/execution/plan/execute_step.py", line 482, in core_dagster_event_sequence_for_step
    for evt in _type_check_and_store_output(step_context, user_event):
  File "/usr/local/lib/python3.8/site-packages/dagster/_core/execution/plan/execute_step.py", line 530, in _type_check_and_store_output
    for output_event in _type_check_output(step_context, step_output_handle, output, version):
  File "/usr/local/lib/python3.8/site-packages/dagster/_core/execution/plan/execute_step.py", line 387, in _type_check_output
    yield DagsterEvent.step_output_event(
  File "/usr/local/lib/python3.8/site-packages/dagster/_core/events/__init__.py", line 825, in step_output_event
    return DagsterEvent.from_step(
  File "/usr/local/lib/python3.8/site-packages/dagster/_core/events/__init__.py", line 430, in from_step
    log_step_event(step_context, event)
  File "/usr/local/lib/python3.8/site-packages/dagster/_core/events/__init__.py", line 309, in log_step_event
    step_context.log.log_dagster_event(
  File "/usr/local/lib/python3.8/site-packages/dagster/_core/log_manager.py", line 406, in log_dagster_event
    self.log(level=level, msg=msg, extra={DAGSTER_META_KEY: dagster_event})
  File "/usr/local/lib/python3.8/site-packages/dagster/_core/log_manager.py", line 421, in log
    self._log(level, msg, args, **kwargs)
  File "/usr/local/lib/python3.8/logging/__init__.py", line 1589, in _log
    self.handle(record)
  File "/usr/local/lib/python3.8/logging/__init__.py", line 1599, in handle
    self.callHandlers(record)
  File "/usr/local/lib/python3.8/logging/__init__.py", line 1661, in callHandlers
    hdlr.handle(record)
  File "/usr/local/lib/python3.8/logging/__init__.py", line 954, in handle
    self.emit(record)
  File "/usr/local/lib/python3.8/site-packages/dagster/_core/log_manager.py", line 287, in emit
    handler.handle(dagster_record)
  File "/usr/local/lib/python3.8/logging/__init__.py", line 954, in handle
    self.emit(record)
  File "/usr/local/lib/python3.8/site-packages/dagster/_core/instance/__init__.py", line 222, in emit
    self._instance.handle_new_event(event)
  File "/usr/local/lib/python3.8/site-packages/dagster/_core/instance/__init__.py", line 2136, in handle_new_event
    self._event_storage.store_event(event)
  File "/usr/local/lib/python3.8/site-packages/dagster_cloud/storage/event_logs/storage.py", line 425, in store_event
    self._execute_query(
  File "/usr/local/lib/python3.8/site-packages/dagster_cloud/storage/event_logs/storage.py", line 303, in _execute_query
    res = self._graphql_client.execute(
  File "/usr/local/lib/python3.8/site-packages/dagster_cloud_cli/core/graphql_client.py", line 127, in execute
    raise GraphQLStorageError(str(e)) from e

The above exception was caused by the following exception:

requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='figment.agent.dagster.cloud', port=443): Read timed out. (read timeout=60)
  File "/usr/local/lib/python3.8/site-packages/dagster_cloud_cli/core/graphql_client.py", line 79, in execute
    return self._execute_retry(query, variable_values, headers)
  File "/usr/local/lib/python3.8/site-packages/dagster_cloud_cli/core/graphql_client.py", line 155, in _execute_retry
    response = <http://self._session.post|self._session.post>(
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 532, in send
    raise ReadTimeout(e, request=request)
The above exception occurred during handling of the following exception:
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='figment.agent.dagster.cloud', port=443): Read timed out. (read timeout=60)
  File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 798, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 714, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 468, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 357, in _raise_timeout
    raise ReadTimeoutError(

The above exception occurred during handling of the following exception:

socket.timeout: The read operation timed out
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 466, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 461, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/lib/python3.8/http/client.py", line 1348, in getresponse
    response.begin()
  File "/usr/local/lib/python3.8/http/client.py", line 316, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.8/http/client.py", line 277, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/lib/python3.8/ssl.py", line 1274, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/local/lib/python3.8/ssl.py", line 1132, in read
    return self._sslobj.read(len, buffer)
The above exception occurred during handling of the following exception:
TypeError: getresponse() got an unexpected keyword argument 'buffering'
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 457, in _make_request
    httplib_response = conn.getresponse(buffering=True)

With a serverless deployment, is it possible add the underlying machine to a VPN network? My thought is that I could install the necessary packages on build then register and connect via the VPNs API. Is this something Dagster Cloud is amenable to?

I’ve written some unit tests that need a connection to a live Snowflake instance. I’ve got a dev/test database in Snowflake where I can run these tests. However, I’m trying to figure out a good pattern to run these tests in my CI pipeline. One option would be to add my test Snowflake secrets to the CI environment (gitlab) and run them there. But that would mean I’d have to maintain these secrets in two places: 1st in the CI environment, and 2nd in dagster serverless environment variables. I’m wondering if anyone has incorporated running pytest in a branch deployment and feeding the results back to CI? Is that a reasonable pattern? Any gotchas? Am I overthinking this and should just deal with managing my secrets in two places?

Is there any way to execute arbitrary python within the context of a dagster branch deployment? I’ve got an error I’m trying to debug and it seems like it has something to do with a difference between my local dev environment and the branch deployement. The only way I can test it now is to make changes, deploy them, and check logs, but that cycle takes about 10 minutes and it would be better if I could iterate faster. If I had a shell in a container that executes the processes, that would be ideal

What is the IP address range for non-isolated runs? appears to be different than what is stated here

https://docs.dagster.io/dagster-cloud/deployment/serverless#whitelisting-dagsters-ip-addresses

Can someone point me to the

latest

version of the Branch Deployment and Prod deployment Github Actions of Dagster Serverless? The reason I asked is that I found this repo and it seems like it’s being deprecated in favor of the ones in this repo. So that’s fine.

This Action is being phased out in favor of our new set of GitHub Actions, for more information see our Hybrid and Serverless quickstart repos.

But then if you look at the dagster-cloud-serverless-quickstart repo, it doesn’t have the same version of branch deployment and prod deploy as this one but has been updated more recently (4 months instead of 1 year). TLDR; Which repo as the most updated Github Actions script for Dagster Serverless option 1 or option 2.

What''s the best way to install a private python module hosted on Github inside Dagster Cloud?

Is it possible to set Dagster Cloud to non-US region?

Anyone else experiencing issues with testing sensors in branch deployments? 🤔 I repeatedly get the

dagster._core.errors.DagsterUserCodeUnreachableError

which tells you to just try again later. It's been nearly a day of waiting, re-deploying, etc., to no avail. I've created a bug ticket here just in case.