https://dagster.io/ logo
#dagster-cloud
Title
# dagster-cloud
r

Ricardo Da Silva

03/12/2024, 2:41 PM
Hello, we are having issues running scheduled jobs in dagster cloud
Copy code
dagster_cloud_cli.core.errors.GraphQLStorageError: Error in GraphQL response: [{'message': 'Internal Server Error (Trace ID: 8637097647672965661)', 'locations': [{'line': 22, 'column': 13}], 'path': ['eventLogs', 'getEventRecords']}]
But we haven’t really changed anything in dagster. Any ideas of what might be going on? Could you guide me to where I could create a ticket? (full error in thread)
🤖 1
Full error: dagster_cloud_cli.core.errors.GraphQLStorageError: Error in GraphQL response: [{‘message’: ‘Internal Server Error (Trace ID: 8637097647672965661)’, ‘locations’: [{‘line’: 22, ‘column’: 13}], ‘path’: [‘eventLogs’, ‘getEventRecords’]}] File “/venvs/a964b4688ab8/lib/python3.8/site-packages/dagster/_core/execution/plan/execute_plan.py”, line 275, in dagster_event_sequence_for_step for step_event in check.generator(step_events): File “/venvs/a964b4688ab8/lib/python3.8/site-packages/dagster/_core/execution/plan/execute_step.py”, line 433, in core_dagster_event_sequence_for_step step_context.fetch_external_input_asset_version_info() File “/venvs/a964b4688ab8/lib/python3.8/site-packages/dagster/_core/execution/context/system.py”, line 976, in fetch_external_input_asset_version_info self._fetch_input_asset_version_info(key) File “/venvs/a964b4688ab8/lib/python3.8/site-packages/dagster/_core/execution/context/system.py”, line 984, in _fetch_input_asset_version_info event = self._get_input_asset_event(key) File “/venvs/a964b4688ab8/lib/python3.8/site-packages/dagster/_core/execution/context/system.py”, line 1037, in _get_input_asset_event event = self.instance.get_latest_data_version_record(key) File “/venvs/a964b4688ab8/lib/python3.8/site-packages/dagster/_core/instance/__init__.py”, line 2801, in get_latest_data_version_record observations = self.get_event_records( File “/venvs/a964b4688ab8/lib/python3.8/site-packages/dagster/_utils/__init__.py”, line 666, in inner return func(*args, **kwargs) File “/venvs/a964b4688ab8/lib/python3.8/site-packages/dagster/_core/instance/__init__.py”, line 1910, in get_event_records return self._event_storage.get_event_records(event_records_filter, limit, ascending) File “/venvs/a964b4688ab8/lib/python3.8/site-packages/dagster_cloud/storage/event_logs/storage.py”, line 500, in get_event_records res = self._execute_query( File “/venvs/a964b4688ab8/lib/python3.8/site-packages/dagster_cloud/storage/event_logs/storage.py”, line 303, in _execute_query res = self._graphql_client.execute( File “/venvs/a964b4688ab8/lib/python3.8/site-packages/dagster_cloud_cli/core/graphql_client.py”, line 147, in execute raise GraphQLStorageError(str(e)) from e The above exception was caused by the following exception: dagster_cloud_cli.core.errors.GraphQLStorageError: Error in GraphQL response: [{‘message’: ‘Internal Server Error (Trace ID: 8637097647672965661)’, ‘locations’: [{‘line’: 22, ‘column’: 13}], ‘path’: [‘eventLogs’, ‘getEventRecords’]}] File “/venvs/a964b4688ab8/lib/python3.8/site-packages/dagster_cloud_cli/core/graphql_client.py”, line 79, in execute return self._execute_retry(query, variable_values, headers) File “/venvs/a964b4688ab8/lib/python3.8/site-packages/dagster_cloud_cli/core/graphql_client.py”, line 191, in _execute_retry raise GraphQLStorageError(f”Error in GraphQL response: {result[‘errors’]}“)
m

Maxime Armstrong

03/12/2024, 2:48 PM
Hey Ricardo - did you upgrade to a new Dagster version?
if you want to open an ticket, you can open one in the Dagster repo under the issue tab by clicking on “New issue”, see here
r

Ricardo Da Silva

03/12/2024, 2:50 PM
@Maxime Armstrong Thanks for the response. We haven’t made any changes that I know of, so the version should be the same. Thanks for the ticket place, I ll also post there
This is on dagster cloud so it should be that version
m

Maxime Armstrong

03/12/2024, 3:09 PM
Ok thanks - we are looking into the issue as we speak
r

Ricardo Da Silva

03/12/2024, 3:09 PM
🙏 appreciate it
d

daniel

03/12/2024, 4:41 PM
Hi Ricardo - we've made a change on our side that we believe should help with this error. Can you confirm whether or not you're still seeing the problem?
r

Ricardo Da Silva

03/12/2024, 4:41 PM
@daniel Let me re run the job and check
@daniel Had a successful run of the job. Will monitor tomorrow that it behaves properly on scheduled run, but for the moment looks good now
👍 1
d

daniel

03/12/2024, 4:53 PM
Got it - apologies for the disruption there
8 Views