https://dagster.io/ logo
#dagster-support
Title
# dagster-support
t

Tom Reilly

07/14/2022, 6:37 PM
What's the best way to orchestrate across python environments? We use the
EcsRunLauncher
and some of our jobs have legacy components that need to run using a different image than the one used by the task definition configured to
EscRunLauncher
. Is there a way to make a specific op use a different image so that our job could look something like:
Copy code
configured op A (default ecs image) ---> configured op B (legacy image) ---> configured op C (default ecs image)
dagster bot responded by community 1
our run launcher is configured like
Copy code
run_launcher:
  module: dagster_aws.ecs
  class: EcsRunLauncher
  config:
    task_definition: prod_dagster_task
    container_name: prod_dagster_container
z

Zach

07/14/2022, 6:47 PM
the way I've accomplished this (in a slightly different context, basically per op multi-cloud execution) is using a custom StepLauncher. But this is far from trivial and requires quite a bit of code and digging into the internals of Dagster execution. There's a discussion on how to do so here. I'd be curious if people have other ideas. alternatively you could just have your op that is dependent on the legacy image just submit a new ecs run with the legacy image and wait for the results, but you lose some of the visibility / traceability of what's going on in the legacy image unless you do some extra work to scrape cloudwatch logs and whatnot
👀 1
2 Views