19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Channels
dagster-de
dagster-cube
gigs-freelance
dagster-wandb
dagster-machine-learning
dagster-bigquery
tools
dagster-dask
dagster-snowflake
dagster-releases
data-platform-design
events
dagster-serverless
dagster-noteable
introductions
dagster-airflow
dagster-cloud
random
dagster-dbt
faq-read-me-before-posting
dagster-airbyte
dagster-spatial
dagster-support
announcements
github-discussions
豆瓣酱帮
dagster-feedback
dagstereo
dagster-ecs
dagster-kubernetes
jobs
dagster-showcase
Title
c
Charles Lariviere
07/20/2021, 8:53 PMWe re-deployed Dagster from Kubernetes to AWS ECS (with the same Postgres database) and are now getting unexpected GraphQL errors when looking at the “Runs” tab. Could that be related to the fact that both deployments were running in parallel with the same database for a few hours?
(See thread for error message)
Operation name: RunsRootQuery
Message: list index out of range
Path: ["pipelineRunsOrError","results",1,"canTerminate"]
Locations: [{"line":36,"column":3}]
Stack Trace:
File "/usr/local/lib/python3.8/site-packages/graphql/execution/executor.py", line 452, in resolve_or_error
return executor.execute(resolve_fn, source, info, **args)
File "/usr/local/lib/python3.8/site-packages/graphql/execution/executors/sync.py", line 16, in execute
return fn(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/dagster_graphql/schema/pipelines/pipeline.py", line 274, in resolve_canTerminate
return graphene_info.context.instance.run_coordinator.can_cancel_run(self.run_id)
File "/usr/local/lib/python3.8/site-packages/dagster/core/run_coordinator/queued_run_coordinator.py", line 107, in can_cancel_run
return self._instance.run_launcher.can_terminate(run_id)
File "/usr/local/lib/python3.8/site-packages/dagster_aws/ecs/launcher.py", line 148, in can_terminate
status = self.ecs.describe_tasks(tasks=[arn], cluster=cluster)["tasks"][0]["lastStatus"]j
jordan
07/20/2021, 9:01 PMHi @Charles Lariviere - I assume this is the same as https://github.com/dagster-io/dagster/issues/4377?
c
Charles Lariviere
07/20/2021, 9:03 PMHmm, it sounds similar, but in our case this is only happening on the
All RunsQueuedIn ProgressI thought it might be related to the fact that we boot up a new Postgres database for our ECS deployment from a snapshot of the other Postgres database we used on the kubernetes deployment. We’re trying to re-boot a new one from a newer snapshot 🤞
j
jordan
07/20/2021, 9:05 PMAh, interesting. I thought Matthias might also be part of your organization.
Still, good data point - now that it’s happened twice 😅
Let me dig in a bit.
🙏 1
What version of Dagster?
c
Charles Lariviere
07/20/2021, 9:05 PM0.11.15
👍 1
I’ll also clarify that both deployments were not running on the same database — that was my mistake. We actually spin up a new one from a snapshot for the ECS deployment.
j
jordan
07/20/2021, 9:11 PM👍 my hope is that the database cutover is a red herring - but we’ll see
👍 1
What does your dagit task’s TaskRole look like?
And perhaps a silly question, but your dagit task is running in the same ECS cluster as all of your other tasks, right?
c
Charles Lariviere
07/20/2021, 9:34 PMWhat does your dagit task’s TaskRole look like?Our TaskRole is currently none — should it be set to something?
And perhaps a silly question, but your dagit task is running in the same ECS cluster as all of your other tasks, right?That’s right! 😄
j
jordan
07/20/2021, 9:38 PMPerfect - that’s what I suspected. I’m about to have a change up for review that should fix this for this Thusrday’s 0.12.3 release.
In short, dagit needs permission to describe ECS tasks 😄. If you’re using the reference deployment for ECS (or your own deployment modeled off of it), we weren’t including those permissions.
I’ll tag you once I have a change up.
c
Charles Lariviere
07/20/2021, 9:39 PMAwesome — thank you for the quick resolution! 🙏
j
jordan
07/20/2021, 9:53 PMhttps://dagster.phacility.com/D8977 and https://dagster.phacility.com/D8978
I think those two should fix things. I haven’t quite reasoned through why
self.ecs.describe_tasks(tasks=[arn], cluster=cluster)tasksfailures🙌 1