19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Is there a way to re-execute a run from failure vi...
# ask-communityc
Charlie Bini
05/10/2023, 9:11 PMIs there a way to re-execute a run from failure via the
run_failure_sensorRunRequest Copy code
@run_failure_sensor
def run_execution_interrupted_sensor(context: RunFailureSensorContext):
run_requests = []
for event in context.get_step_failure_events():
if event.event_specific_data.error_source == ErrorSource.INTERRUPT:
...
run_requests.append(RunRequest(...))
return SensorResult(run_requests=run_requests)Charlie Bini
05/10/2023, 9:12 PMtrying to build a smarter way to combat k8s killing run pods. Usually, this happens when GKE changes the nodes under the hood
j
Jordan
05/10/2023, 9:34 PMHi! I personally use graphql to launch runs from
run_failure_sensors
sean
05/11/2023, 12:19 PMHi Charlie, I don’t think it’s possible to “directly” reexecute from a run failure sensor, but I’ve reached out to the team for confirmation.
To elaborate on what Jordan is saying, you could use the GQL python client as a workaround: https://docs.dagster.io/concepts/dagit/graphql-client#graphql-python-client
c
Charlie Bini
05/11/2023, 2:46 PMnice yeah I hadn't considered graphql
s
sean
05/11/2023, 3:04 PMHey Charlie, I’ve confirmed you can’t reexecute from failure via the sensor. However, you might be able to avoid the need for this with Run Retries
c
Charlie Bini
05/11/2023, 7:14 PM@sean should the graphql client work for dagster cloud?
Charlie Bini
05/11/2023, 7:15 PMI'm trying a simple
get_run_statusCharlie Bini
05/11/2023, 7:43 PMchanged some stuff and I'm seeing
Copy code
gql.transport.exceptions.TransportServerError: 401 Client Error: Unauthorized for url: <https://kipptaf.dagster.cloud/graphql>Charlie Bini
05/11/2023, 7:44 PMhere's my code
Copy code
@run_failure_sensor
def run_execution_interrupted_sensor(context: RunFailureSensorContext):
client = DagsterGraphQLClient(hostname="kipptaf.dagster.cloud")
for event in context.get_step_failure_events():
run_status = client.get_run_status(event.logging_tags["run_id"])
<http://context.log.info|context.log.info>(run_status)Charlie Bini
05/11/2023, 7:46 PM@Jordan if you have any snippets you can share, that'd be appreciated
Charlie Bini
05/11/2023, 7:58 PMdifference between the two is the port:
• no port = 401
• port 3000 = hanging
j
Jordan
05/11/2023, 8:08 PMI don't use the dagster cloud version so I can't really help you with that.
My code looks like this:
Copy code
url = "<http://127.0.0.1:3000/graphql>"
payload = {"query": graphql_query, "variables": variables}
headers = {"Content-Type": "application/json"}
response = requests.request("POST", url=url, json=payload, headers=headers)d
Daniel Gafni
06/05/2023, 8:06 PMHey @sean, so how to authorize the GraphQL client for Dagster Cloud? The docs don't seem to cover that.
s
sean
06/09/2023, 10:20 PMDoes this help? https://github.com/dagster-io/dagster/discussions/7772
d
Daniel Gafni
06/09/2023, 10:25 PMYes, thank you
2 Views