https://dagster.io/ logo
Join Slack
Powered by
Hey guys! I'm trying to trigger ECS task from loca...
# ask-community
s
Hey guys! I'm trying to trigger ECS task from local dagster installation. I've observed that the ECS task gets triggered but the job in dagster UI remains in "Starting" stage forever..even after the ECS task gets terminated. Has anyone came across this issue? Could someone please help with this? Thanks
d
Hi Sanjay - I would suggest checking the logs of the ECS task in the ECS console to see why it was terminated
you can use this feature in Dagster to move runs into failed in dagster if the task gets stuck starting u0p: https://docs.dagster.io/deployment/run-monitoring#run-start-timeouts
s
The task is getting completed successfully in ES with status code 0. But, it remains in Starting stage in Dagster UI. I also tried with Run monitoring..but it just marks it as "failed" after the configurable "start_timeout_seconds". Somehow, it's not transitioning to STARTED state.
As per the doc above, it states that - When Dagster launches a run, the run stays in STARTING status until the run worker spins up and marks the run as STARTED For ECSLauncher, not sure when would the run worker spin up? Or do we need to add anything in dagster.yaml to spin up the run worker?
d
Are there any clues in the logs for the task? Can you post to the json of the task definition?
s
ECS task is working correctly as expected..no issues with the ECS task that is triggered from dagster run from local.
Just that in the dagster UI somehow the state isn't transitioning to "STARTED" state
d
You’re using the EcsRunlauncher?
Can you share the task definition json?
and maybe your dagster.yaml / how you have things configured to launch the runs?
s
yes, I am using the ECSRunLauncher
Copy code
run_launcher:
  module: "dagster_aws.ecs"
  class: "EcsRunLauncher"
  config:
    task_definition: "task_definition_arn"
    container_name: "container_arn"
    use_current_ecs_task_config: False
This is the dagster.yaml file
d
Can you share the logs from the ECS task?
Er wait
Is that the whole dagster.yaml?
s
I am just printing hello world for now
d
It needs to include some kind of storage - usually postgres
s
in the log
default is sqlite in the same working directory..correct?
d
That’s right - which isn’t available in the ecs task since it’s on a different machine
Which explains what you’re seeing - the task isn’t writing to your database
It’s writing to its own database on the local filesystem in the task
s
The ECS task job is to just print hello world.. My idea of using dagster from local is to trigger that ECS task
d
Got it - that’s not a feature that we currently support unfortunately
s
My expectation is that local dagster worker should trigger the ECS task and dagster local should monitor that task and on ECS task completion..it should stop the dagster run
d
It has to be a dagster run / job, and it has to have access to dagster storage
We don’t support running arbitrary non dagster code in an Ecs task via the Ecs run launcher
You could have an op that spins up a task using the ecs api though?
And then launch the job normally locally
s
Oh, so you expect dagster to be in any other ECS cluster and from there, I can trigger ECS tasks in another cluster?
d
It doesn’t have to be in an ecs cluster to use the ecs api
Ecs has a python api that can be called from anywhere as long as you have credentials https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs/client/run_task.html
s
Got it..thanks
5 Views
Open in Slack
PreviousNext
Powered by