Joel Olazagasti
08/07/2023, 7:52 PMdbt deps
to install my dbt packages? We're currently on the docker deployment with a post install scriptTodd de Quincey
08/12/2023, 5:30 PMStefan Adelbert
08/18/2023, 2:53 AMSon Do
08/18/2023, 7:19 PMbotocore.exceptions.ClientError: An error occurred (ThrottlingException) when calling the ListTasks operation: Rate exceeded
File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1256, in _reconcile
self._wait_for_new_multipex_server(
File "/dagster-cloud/dagster_cloud/workspace/ecs/launcher.py", line 414, in _wait_for_new_multipex_server
task_arn = self.client.wait_for_new_service(
File "/dagster-cloud/dagster_cloud/workspace/ecs/client.py", line 497, in wait_for_new_service
return self.check_service_has_running_task(
File "/dagster-cloud/dagster_cloud/workspace/ecs/client.py", line 557, in check_service_has_running_task
running = self.ecs.list_tasks(
File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 535, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 980, in _make_api_call
raise error_class(parsed_response, operation_name
Kyle Montag
08/21/2023, 1:47 PMSébastien Duguay
08/23/2023, 5:23 PMCopy
botocore.exceptions.ClientError: An error occurred (ThrottlingException) when calling the DescribeTasks operation: Rate exceeded
File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1256, in _reconcile
self._wait_for_new_multipex_server(
File "/dagster-cloud/dagster_cloud/workspace/ecs/launcher.py", line 414, in _wait_for_new_multipex_server
task_arn = self.client.wait_for_new_service(
File "/dagster-cloud/dagster_cloud/workspace/ecs/client.py", line 497, in wait_for_new_service
return self.check_service_has_running_task(
File "/dagster-cloud/dagster_cloud/workspace/ecs/client.py", line 587, in check_service_has_running_task
task = self.ecs.describe_tasks(
File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 535, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 980, in _make_api_call
raise error_class(parsed_response, operation_name)
Todd de Quincey
09/01/2023, 6:04 PMAWS_REGION
env var being overridden in ServerlessJacob Marcil
09/07/2023, 3:24 PMretry_policy
of an asset?
I need to query API that are not very reliable (The data can show up with 1 hour to 3 days of delay), and was wondering if I configure my retry_policy
to something like 1 or 2 days, would I pay for those days while doing nothing since my run
is technically in progress?Son Do
09/18/2023, 3:06 PMDavid Merritt
09/18/2023, 11:40 PMCharles
09/19/2023, 8:29 AMJacob Marcil
09/19/2023, 9:41 PMpartitions
to keep track of them. T
hat would create something around 500k partitions for that specific asset growing at a rate of 1000 partitions per day. Would that highly increase the cost of Dagster Serverless even if the compute time if very small?David Merritt
09/21/2023, 12:08 AMManish Khatri
09/21/2023, 10:42 AMAllocating compute resources to ensure run isolation. This can take a few minutes.
Still waiting for compute resources to spin up.
Still waiting for compute resources to spin up.
...
Still waiting for compute resources to spin up.
Agent b48510b6 detected that run worker failed.
In terms of a better experience around this, I believe that Dagster should catch this specific exception and have some type of retry mechanism in place. It feels lazy that Dagster tried once, encountered this failure and then gave up so quickly. If our impacted job was a very important daily job, it could have been much more disruptive.Anthony Yim
09/21/2023, 10:50 PMGregory Ivanyutin
10/10/2023, 11:48 PM23:35:51 Encountered an error:
Parsing Error
Env var required but not provided: 'SNOWFLAKE_ACCOUNT'
Error: Process completed with exit code 2.
Got my envs like account: "{{ env_var('SNOWFLAKE_ACCOUNT') }}" in profiles.yml. Serverless job fails on Parse dbt project stepManish Khatri
10/12/2023, 10:42 AMStep custom_dbt_run_op finished without success or failure event. Downstream steps will not execute.
However you can clearly see in the log output that it was successful blob confounded
Is this a bug? Has it been fixed/resolved in a newer version?
Current Dagster version 1.4.6
Jacob Marcil
10/12/2023, 12:06 PMJacob Marcil
10/12/2023, 3:37 PMoscrypto.errors.LibraryNotFoundError: Error detecting the version of libcrypto
Charles Couble
10/17/2023, 12:55 PMJacob Marcil
10/17/2023, 8:00 PMimport a dbt project
using the feature in Deployment -> Add code location -> Import a dbt project
.
And I’m not sure what to do. My security team tells me that all grants should be available for Dagster to see m DBT repo, but Dagster Cloud Serverless is not able to find it.
Is there something that need to be in that repo in order for Dagster Cloud Serverless to see it? Or a basic DBT Repository is fine?
Or do we need a DBT repo wrap with Dagster in order to be able to import it?
Thank you very much.
Have a great day.Jacob Marcil
10/18/2023, 6:55 PMRead Timed out
error while building DBT models with Dagster Cloud Serverless on Snowflake.
Is that a known issue or should I create a Dagster issue for it?
Here’s the complete stack trace.
dagster_cloud_cli.core.errors.GraphQLStorageError: HTTPSConnectionPool(host='figment.agent.dagster.cloud', port=443): Read timed out. (read timeout=60)
File "/usr/local/lib/python3.8/site-packages/dagster/_core/execution/plan/execute_plan.py", line 273, in dagster_event_sequence_for_step
for step_event in check.generator(step_events):
File "/usr/local/lib/python3.8/site-packages/dagster/_core/execution/plan/execute_step.py", line 482, in core_dagster_event_sequence_for_step
for evt in _type_check_and_store_output(step_context, user_event):
File "/usr/local/lib/python3.8/site-packages/dagster/_core/execution/plan/execute_step.py", line 530, in _type_check_and_store_output
for output_event in _type_check_output(step_context, step_output_handle, output, version):
File "/usr/local/lib/python3.8/site-packages/dagster/_core/execution/plan/execute_step.py", line 387, in _type_check_output
yield DagsterEvent.step_output_event(
File "/usr/local/lib/python3.8/site-packages/dagster/_core/events/__init__.py", line 825, in step_output_event
return DagsterEvent.from_step(
File "/usr/local/lib/python3.8/site-packages/dagster/_core/events/__init__.py", line 430, in from_step
log_step_event(step_context, event)
File "/usr/local/lib/python3.8/site-packages/dagster/_core/events/__init__.py", line 309, in log_step_event
step_context.log.log_dagster_event(
File "/usr/local/lib/python3.8/site-packages/dagster/_core/log_manager.py", line 406, in log_dagster_event
self.log(level=level, msg=msg, extra={DAGSTER_META_KEY: dagster_event})
File "/usr/local/lib/python3.8/site-packages/dagster/_core/log_manager.py", line 421, in log
self._log(level, msg, args, **kwargs)
File "/usr/local/lib/python3.8/logging/__init__.py", line 1589, in _log
self.handle(record)
File "/usr/local/lib/python3.8/logging/__init__.py", line 1599, in handle
self.callHandlers(record)
File "/usr/local/lib/python3.8/logging/__init__.py", line 1661, in callHandlers
hdlr.handle(record)
File "/usr/local/lib/python3.8/logging/__init__.py", line 954, in handle
self.emit(record)
File "/usr/local/lib/python3.8/site-packages/dagster/_core/log_manager.py", line 287, in emit
handler.handle(dagster_record)
File "/usr/local/lib/python3.8/logging/__init__.py", line 954, in handle
self.emit(record)
File "/usr/local/lib/python3.8/site-packages/dagster/_core/instance/__init__.py", line 222, in emit
self._instance.handle_new_event(event)
File "/usr/local/lib/python3.8/site-packages/dagster/_core/instance/__init__.py", line 2136, in handle_new_event
self._event_storage.store_event(event)
File "/usr/local/lib/python3.8/site-packages/dagster_cloud/storage/event_logs/storage.py", line 425, in store_event
self._execute_query(
File "/usr/local/lib/python3.8/site-packages/dagster_cloud/storage/event_logs/storage.py", line 303, in _execute_query
res = self._graphql_client.execute(
File "/usr/local/lib/python3.8/site-packages/dagster_cloud_cli/core/graphql_client.py", line 127, in execute
raise GraphQLStorageError(str(e)) from e
The above exception was caused by the following exception:
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='figment.agent.dagster.cloud', port=443): Read timed out. (read timeout=60)
File "/usr/local/lib/python3.8/site-packages/dagster_cloud_cli/core/graphql_client.py", line 79, in execute
return self._execute_retry(query, variable_values, headers)
File "/usr/local/lib/python3.8/site-packages/dagster_cloud_cli/core/graphql_client.py", line 155, in _execute_retry
response = <http://self._session.post|self._session.post>(
File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 637, in post
return self.request("POST", url, data=data, json=json, **kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 532, in send
raise ReadTimeout(e, request=request)
The above exception occurred during handling of the following exception:
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='figment.agent.dagster.cloud', port=443): Read timed out. (read timeout=60)
File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 798, in urlopen
retries = retries.increment(
File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 550, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 770, in reraise
raise value
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 714, in urlopen
httplib_response = self._make_request(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 468, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 357, in _raise_timeout
raise ReadTimeoutError(
The above exception occurred during handling of the following exception:
socket.timeout: The read operation timed out
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 466, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 461, in _make_request
httplib_response = conn.getresponse()
File "/usr/local/lib/python3.8/http/client.py", line 1348, in getresponse
response.begin()
File "/usr/local/lib/python3.8/http/client.py", line 316, in begin
version, status, reason = self._read_status()
File "/usr/local/lib/python3.8/http/client.py", line 277, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/local/lib/python3.8/socket.py", line 669, in readinto
return self._sock.recv_into(b)
File "/usr/local/lib/python3.8/ssl.py", line 1274, in recv_into
return self.read(nbytes, buffer)
File "/usr/local/lib/python3.8/ssl.py", line 1132, in read
return self._sslobj.read(len, buffer)
The above exception occurred during handling of the following exception:
TypeError: getresponse() got an unexpected keyword argument 'buffering'
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 457, in _make_request
httplib_response = conn.getresponse(buffering=True)
Brandon Freeman
10/23/2023, 6:13 PMSterling Paramore
10/23/2023, 9:57 PMSterling Paramore
10/25/2023, 7:30 PMBrent Shulman
10/30/2023, 2:50 PMhttps://docs.dagster.io/dagster-cloud/deployment/serverless#whitelisting-dagsters-ip-addresses
Jacob Marcil
11/02/2023, 2:36 PMlatest
version of the Branch Deployment and Prod deployment Github Actions of Dagster Serverless?
The reason I asked is that I found this repo and it seems like it’s being deprecated in favor of the ones in this repo. So that’s fine.
This Action is being phased out in favor of our new set of GitHub Actions, for more information see our Hybrid and Serverless quickstart repos.But then if you look at the dagster-cloud-serverless-quickstart repo, it doesn’t have the same version of branch deployment and prod deploy as this one but has been updated more recently (4 months instead of 1 year). TLDR; Which repo as the most updated Github Actions script for Dagster Serverless option 1 or option 2.
Charles
11/08/2023, 5:02 PMCharles
11/10/2023, 12:34 AMBrandon Freeman
11/17/2023, 3:59 PMdagster._core.errors.DagsterUserCodeUnreachableError
which tells you to just try again later. It's been nearly a day of waiting, re-deploying, etc., to no avail.
I've created a bug ticket here just in case.