19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Channels
announcements
dagster-airbyte
dagster-airflow
dagster-bigquery
dagster-cloud
dagster-cube
dagster-dask
dagster-dbt
dagster-de
dagster-ecs
dagster-feedback
dagster-kubernetes
dagster-machine-learning
dagster-noteable
dagster-releases
dagster-serverless
dagster-showcase
dagster-snowflake
dagster-spatial
dagster-support
dagster-wandb
dagstereo
data-platform-design
events
faq-read-me-before-posting
gigs-freelance
github-discussions
introductions
jobs
random
tools
豆瓣酱帮
Title
c
Charlie Bini
05/10/2023, 9:11 PMIs there a way to re-execute a run from failure via the
run_failure_sensorRunRequest@run_failure_sensor
def run_execution_interrupted_sensor(context: RunFailureSensorContext):
run_requests = []
for event in context.get_step_failure_events():
if event.event_specific_data.error_source == ErrorSource.INTERRUPT:
...
run_requests.append(RunRequest(...))
return SensorResult(run_requests=run_requests)trying to build a smarter way to combat k8s killing run pods. Usually, this happens when GKE changes the nodes under the hood
j
Jordan
05/10/2023, 9:34 PMHi! I personally use graphql to launch runs from
run_failure_sensors
sean
05/11/2023, 12:19 PMHi Charlie, I don’t think it’s possible to “directly” reexecute from a run failure sensor, but I’ve reached out to the team for confirmation.
To elaborate on what Jordan is saying, you could use the GQL python client as a workaround: https://docs.dagster.io/concepts/dagit/graphql-client#graphql-python-client
c
Charlie Bini
05/11/2023, 2:46 PMnice yeah I hadn't considered graphql
s
sean
05/11/2023, 3:04 PMHey Charlie, I’ve confirmed you can’t reexecute from failure via the sensor. However, you might be able to avoid the need for this with Run Retries
c
Charlie Bini
05/11/2023, 7:14 PM@sean should the graphql client work for dagster cloud?
I'm trying a simple
get_run_statuschanged some stuff and I'm seeing
gql.transport.exceptions.TransportServerError: 401 Client Error: Unauthorized for url: <https://kipptaf.dagster.cloud/graphql>here's my code
@run_failure_sensor
def run_execution_interrupted_sensor(context: RunFailureSensorContext):
client = DagsterGraphQLClient(hostname="kipptaf.dagster.cloud")
for event in context.get_step_failure_events():
run_status = client.get_run_status(event.logging_tags["run_id"])
<http://context.log.info|context.log.info>(run_status)@Jordan if you have any snippets you can share, that'd be appreciated
difference between the two is the port:
• no port = 401
• port 3000 = hanging
j
Jordan
05/11/2023, 8:08 PMI don't use the dagster cloud version so I can't really help you with that.
My code looks like this:
url = "<http://127.0.0.1:3000/graphql>"
payload = {"query": graphql_query, "variables": variables}
headers = {"Content-Type": "application/json"}
response = requests.request("POST", url=url, json=payload, headers=headers)