19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Hi, we currently use Dagster Cloud and I wanted to...
# dagster-plusm
Manish Khatri
03/01/2023, 4:07 PMHi, we currently use Dagster Cloud and I wanted to know if Dagster supports triggering a job in ECS via the AWS integration (https://docs.dagster.io/_apidocs/libraries/dagster-aws#ecs). The documentation doesn’t really clarify or give much idea on if this is possible, and an example of how?
If it does, does it also get log information and show it within Dagster?
d
daniel
03/01/2023, 4:08 PMHi Manish - is this serverless or hybrid?
m
Manish Khatri
03/01/2023, 4:08 PMHi Daniel, this is serverless. So it’s hosted on dagster owned infra
d
daniel
03/01/2023, 4:09 PMAre you hoping to trigger a task in ECS in a separate ECS cluster that you control? Or launch a task within a dagster owned ECS cluster?
m
Manish Khatri
03/01/2023, 4:09 PMThe former, we would aim to trigger a job in our own cluster )
d
daniel
03/01/2023, 4:10 PMI see - I believe that should be possible using the Python boto3 ECS api in an op that you write - but we don't have built-in support for launching arbitrary ECS tasks in Dagster currently
m
Manish Khatri
03/01/2023, 4:11 PMOk. So that particular integration is only for hybrid solution that is hosted on it’s own ecs cluster?
d
daniel
03/01/2023, 4:13 PMBoth hybrid and serverless launch an ECS task for each run in an ECS cluster - the main difference between serverless and hybrid is that we host the ECS cluster for you
m
Manish Khatri
03/01/2023, 4:14 PMOk thanks for clarifying.
Is there any other way we can trigger some sort of containerised application (docker image) from within dagster serverless that uses some sort if integration dagster already provides? Or would this be a custom thing?
d
daniel
03/01/2023, 4:15 PMRight now that would need to be a custom thing I think, although I could imagine it being a pretty thin layer on top of the boto3 run_task API if you pass in credentials to your own ECS cluster
m
Manish Khatri
03/01/2023, 4:17 PMWould you know off the top if the effort to getting log outputs would be high?
d
daniel
03/01/2023, 4:18 PMI think that would probably involve a call out to the cloudwatch API and printing out the results within the op - doesn't sound too difficult via boto
daniel
03/01/2023, 4:19 PMI think if you already have an ECS cluster that's working for you and you're planning on launching ECS tasks there within serverless runs, then ECS hybrid is also worth considering
m
Manish Khatri
03/01/2023, 4:22 PMWe currently don’t, we are looking at workarounds over wanting to run jobs that process our customer data only within Europe, Dagster servers are running in USA only right now.
Manish Khatri
03/01/2023, 8:55 PMHI Daniel, side question - is it possible for us to tell dagster to run a Docker image within dagster serverless? The image would be available to pull from our own Docker repository .
The question comes from a place where we can separate our own jobs written in Python (or any other language) from the dagster project, but still have it triggered and executed by Dagster.
d
daniel
03/01/2023, 8:55 PMCurrently we need to host any docker images running in serverless in our cluster
daniel
03/01/2023, 8:56 PMBut that's 100% possible in hybrid
m
Manish Khatri
03/01/2023, 8:57 PMThanks 👍 is there any documentation on how to do this? I couldn’t find it online
d
daniel
03/01/2023, 8:58 PMdocumentation on how to run docker images in hybrid?
m
Manish Khatri
03/01/2023, 8:58 PMYup
d
daniel
03/01/2023, 8:59 PMWe have ops that let you run arbitrary containers in k8s: https://docs.dagster.io/_apidocs/libraries/dagster-k8s#dagster_k8s.k8s_job_op and docker: https://docs.dagster.io/_apidocs/libraries/dagster-docker#dagster_docker.docker_container_op - we don't yet have an ecs_task_op but it wouldn't be hard to build one on top of the ECS Python API
m
Manish Khatri
03/01/2023, 9:02 PMTaking a look at
dagster_docker.execute_docker_containerd
daniel
03/01/2023, 9:03 PMIt's not a feature we currently support in serverless unfortunately - running docker containers on some other server that you are hosting may be an option by customizing the host that it uses to connect to the Docker socket, but we don't support running arbitrary docker containers in our serverless cluster
👍 1
m
Manish Khatri
03/01/2023, 9:04 PMAh ok. Thanks for clarifying 🙂
3 Views