19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Hello, we are having issues running scheduled jobs...
# dagster-plusr
Ricardo Da Silva
03/12/2024, 2:41 PMHello, we are having issues running scheduled jobs in dagster cloud
Copy code
dagster_cloud_cli.core.errors.GraphQLStorageError: Error in GraphQL response: [{'message': 'Internal Server Error (Trace ID: 8637097647672965661)', 'locations': [{'line': 22, 'column': 13}], 'path': ['eventLogs', 'getEventRecords']}]🤖 1
Ricardo Da Silva
03/12/2024, 2:41 PMFull error:
dagster_cloud_cli.core.errors.GraphQLStorageError: Error in GraphQL response: [{‘message’: ‘Internal Server Error (Trace ID: 8637097647672965661)’, ‘locations’: [{‘line’: 22, ‘column’: 13}], ‘path’: [‘eventLogs’, ‘getEventRecords’]}]
File “/venvs/a964b4688ab8/lib/python3.8/site-packages/dagster/_core/execution/plan/execute_plan.py”, line 275, in dagster_event_sequence_for_step
for step_event in check.generator(step_events):
File “/venvs/a964b4688ab8/lib/python3.8/site-packages/dagster/_core/execution/plan/execute_step.py”, line 433, in core_dagster_event_sequence_for_step
step_context.fetch_external_input_asset_version_info()
File “/venvs/a964b4688ab8/lib/python3.8/site-packages/dagster/_core/execution/context/system.py”, line 976, in fetch_external_input_asset_version_info
self._fetch_input_asset_version_info(key)
File “/venvs/a964b4688ab8/lib/python3.8/site-packages/dagster/_core/execution/context/system.py”, line 984, in _fetch_input_asset_version_info
event = self._get_input_asset_event(key)
File “/venvs/a964b4688ab8/lib/python3.8/site-packages/dagster/_core/execution/context/system.py”, line 1037, in _get_input_asset_event
event = self.instance.get_latest_data_version_record(key)
File “/venvs/a964b4688ab8/lib/python3.8/site-packages/dagster/_core/instance/__init__.py”, line 2801, in get_latest_data_version_record
observations = self.get_event_records(
File “/venvs/a964b4688ab8/lib/python3.8/site-packages/dagster/_utils/__init__.py”, line 666, in inner
return func(*args, **kwargs)
File “/venvs/a964b4688ab8/lib/python3.8/site-packages/dagster/_core/instance/__init__.py”, line 1910, in get_event_records
return self._event_storage.get_event_records(event_records_filter, limit, ascending)
File “/venvs/a964b4688ab8/lib/python3.8/site-packages/dagster_cloud/storage/event_logs/storage.py”, line 500, in get_event_records
res = self._execute_query(
File “/venvs/a964b4688ab8/lib/python3.8/site-packages/dagster_cloud/storage/event_logs/storage.py”, line 303, in _execute_query
res = self._graphql_client.execute(
File “/venvs/a964b4688ab8/lib/python3.8/site-packages/dagster_cloud_cli/core/graphql_client.py”, line 147, in execute
raise GraphQLStorageError(str(e)) from e
The above exception was caused by the following exception:
dagster_cloud_cli.core.errors.GraphQLStorageError: Error in GraphQL response: [{‘message’: ‘Internal Server Error (Trace ID: 8637097647672965661)’, ‘locations’: [{‘line’: 22, ‘column’: 13}], ‘path’: [‘eventLogs’, ‘getEventRecords’]}]
File “/venvs/a964b4688ab8/lib/python3.8/site-packages/dagster_cloud_cli/core/graphql_client.py”, line 79, in execute
return self._execute_retry(query, variable_values, headers)
File “/venvs/a964b4688ab8/lib/python3.8/site-packages/dagster_cloud_cli/core/graphql_client.py”, line 191, in _execute_retry
raise GraphQLStorageError(f”Error in GraphQL response: {result[‘errors’]}“)
m
Maxime Armstrong
03/12/2024, 2:48 PMHey Ricardo - did you upgrade to a new Dagster version?
Maxime Armstrong
03/12/2024, 2:49 PMif you want to open an ticket, you can open one in the Dagster repo under the issue tab by clicking on “New issue”, see here
r
Ricardo Da Silva
03/12/2024, 2:50 PM@Maxime Armstrong Thanks for the response. We haven’t made any changes that I know of, so the version should be the same.
Thanks for the ticket place, I ll also post there
Ricardo Da Silva
03/12/2024, 2:53 PMThis is on dagster cloud so it should be that version
m
Maxime Armstrong
03/12/2024, 3:09 PMOk thanks - we are looking into the issue as we speak
r
Ricardo Da Silva
03/12/2024, 3:09 PM🙏 appreciate it
d
daniel
03/12/2024, 4:41 PMHi Ricardo - we've made a change on our side that we believe should help with this error. Can you confirm whether or not you're still seeing the problem?
r
Ricardo Da Silva
03/12/2024, 4:41 PM@daniel Let me re run the job and check
Ricardo Da Silva
03/12/2024, 4:52 PM@daniel Had a successful run of the job. Will monitor tomorrow that it behaves properly on scheduled run, but for the moment looks good now
👍 1
d
daniel
03/12/2024, 4:53 PMGot it - apologies for the disruption there
11 Views