What's the best way to orchestrate across python e...
# ask-community
What's the best way to orchestrate across python environments? We use the
and some of our jobs have legacy components that need to run using a different image than the one used by the task definition configured to
. Is there a way to make a specific op use a different image so that our job could look something like:
Copy code
configured op A (default ecs image) ---> configured op B (legacy image) ---> configured op C (default ecs image)
dagster bot responded by community 1
our run launcher is configured like
Copy code
  module: dagster_aws.ecs
  class: EcsRunLauncher
    task_definition: prod_dagster_task
    container_name: prod_dagster_container
the way I've accomplished this (in a slightly different context, basically per op multi-cloud execution) is using a custom StepLauncher. But this is far from trivial and requires quite a bit of code and digging into the internals of Dagster execution. There's a discussion on how to do so here. I'd be curious if people have other ideas. alternatively you could just have your op that is dependent on the legacy image just submit a new ecs run with the legacy image and wait for the results, but you lose some of the visibility / traceability of what's going on in the legacy image unless you do some extra work to scrape cloudwatch logs and whatnot
👀 1