Hi All, On my first implementation of Dagster in a Container on GCP I am getting the following error on subsequent runs:

Unexpected GraphQL error

with the following details: Operation name: InstanceWarningQuery

Message: (sqlite3.OperationalError) disk I/O error
[SQL: SELECT daemon_heartbeats.daemon_type, daemon_heartbeats.daemon_id, daemon_heartbeats.timestamp, daemon_heartbeats.body 
FROM daemon_heartbeats]
(Background on this error at: <https://sqlalche.me/e/14/e3q8>)

Path: ["instance","daemonHealth","allDaemonStatuses"]

Locations: [{"line":30,"column":3}]

Can any one assist with the issue?

do you have access to the container logs for the dagster server?

I do have access to the logs and have just rerun the process to generate fresh logs. This is the final log for the container:

2023-03-09 10:34:02 +0000 - dagster - DEBUG - settlement_model - 1302564d-5be9-43d3-bd79-ad1477eb93ca - prepare_settlement_kenya_model_data - Writing file at: /app/build/tmpvbtm_meu/storage/1302564d-5be9-43d3-bd79-ad1477eb93ca/prepare_settlement_kenya_model_data/result
Exception while writing logger call to event log: (sqlite3.OperationalError) disk I/O error
[SQL: INSERT INTO event_logs (run_id, event, dagster_event_type, timestamp, step_key, asset_key, partition) VALUES (?, ?, ?, ?, ?, ?, ?)]
[parameters: ('1302564d-5be9-43d3-bd79-ad1477eb93ca', '{"__class__": "EventLogEntry", "dagster_event": null, "error_info": null, "level": 40, "message": "", "pipeline_name": "settlement_model", "run_id":  ... (78 characters truncated) ... 044.385932, "user_message": "Step prepare_settlement_kenya_model_data finished without success or failure event. Downstream steps will not execute."}', None, '2023-03-09 10:34:04.385932', None, None, None)]
(Background on this error at: <https://sqlalche.me/e/14/e3q8>)
Exception while writing logger call to event log: (sqlite3.OperationalError) disk I/O error
[SQL: INSERT INTO event_logs (run_id, event, dagster_event_type, timestamp, step_key, asset_key, partition) VALUES (?, ?, ?, ?, ?, ?, ?)]
[parameters: ('1302564d-5be9-43d3-bd79-ad1477eb93ca', '{"__class__": "EventLogEntry", "dagster_event": {"__class__": "DagsterEvent", "event_specific_data": {"__class__": "PipelineFailureData", "error": {" ... (5371 characters truncated) ... ull, "timestamp": 1678358044.402503, "user_message": "Execution of run for \\"settlement_model\\" failed. An exception was thrown during execution."}', 'PIPELINE_FAILURE', '2023-03-09 10:34:04.402503', None, None, None)]
(Background on this error at: <https://sqlalche.me/e/14/e3q8>)

@Adam Borlase did you ever resolve this? I'm getting the same error, thanks.

Hi @Terry Lines, I unfortunately did not, We have hence moved over to Dagtser Cloud to see if it will work better there but I was getting an error on the Container logs as well which did not really help

Ok. Mine was an out of disk space error, if that helps

Hey Adam - I’ve found the issue here. And I’ve let @chris know

Thanks so much @Jun Jo, much appreciated.